Custom responses for resource unavailable errors

ABSTRACT

A proxy server receives from a client device a request for a network resource hosted at an origins server for a domain. The request is received at the proxy server as a result of a DNS request for the domain resolving to the proxy server. The origin server is one of multiple origin servers that belong to different domains and resolve to the proxy server and are owned by different entities. The proxy server requests the network resource from the origin server. The proxy server receives a response from the origin server that indicates that the network resource is unavailable. The proxy server transmits a custom error page to the client device that indicates that the requested resource is unavailable.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 15/495,785filed Apr. 24, 2017, which claims the benefit of application Ser. No.13/078,899, filed Apr. 1, 2011, now U.S. Pat. No. 9,634,994, whichapplication claims the benefit of U.S. Provisional Application No.61/397,721, filed Apr. 1, 2010, both of which are hereby incorporated byreference.

BACKGROUND Field

Embodiments of the invention relate to the field of network services;and more specifically to Internet-based proxy services.

Background

Internet servers, by their nature, are accessible via the Internet andare capable of being compromised and/or attacked. These attacks include,among other things, blog or other comment spam POSTs, SQL injectionPOSTs, cross-site scripting POSTs, denial of service (DoS) attacks,query floods, excessive bandwidth use, or requests that exploit otherknown weakness of the servers. Attacks may be implemented using botnets(or sometimes referred to as bots), which are typically infectedpersonal computers running on home or office networks. The personalcomputers may be infected in a number of ways, for example by visiting asite with malicious code, executing software that installs maliciouscode, etc. The legitimate users of these infected personal computers areoften unaware of the infection and their use in a botnet attack.

Web application firewalls (WAFs), which are either hardware devicesinstalled in a network operator's data center or software that isinstalled on the web server, may monitor traffic routed to the webserver in order to detect and stop potential attacks. Unlike traditionalfirewalls that focus on the network layer, web application firewallsperform deep packet inspection to look for attack signatures at theapplication level.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments of the invention. In the drawings:

FIG. 1 illustrates an exemplary architecture of an Internet-based proxyservice according to one embodiment of the invention;

FIG. 2 is a block diagram illustrating an exemplary request module of aproxy server of the service that processes requests according to oneembodiment of the invention;

FIG. 3 is a block diagram illustrating an exemplary response module ofthe proxy server that processes responses according to one embodiment ofthe invention;

FIGS. 4A-B are flow diagrams illustrating exemplary operations for acustomer to use the service server to register for service according toone embodiment;

FIG. 5 illustrates an exemplary interface provided by the service serverto allow domain owners to enter the information for the DNS zone filerecords according to one embodiment of the invention;

FIG. 6 is a flow diagram illustrating exemplary operations performed bythe service server to assist domain owners in manually entering DNS zonefile information according to one embodiment of the invention;

FIGS. 7A-C illustrate exemplary interfaces for customers to input domainrelated information according to one embodiment of the invention;

FIG. 8 is a flow diagram illustrating exemplary operations performed bya proxy server according to one embodiment of the invention;

FIG. 9 is a flow diagram illustrating exemplary operations to determinewhether a request and/or a visitor is an Internet security threataccording to one embodiment of the invention;

FIGS. 10A-B are exemplary block pages according to one embodiment of theinvention;

FIG. 11 is a flow diagram illustrating exemplary operations fordetermining whether the request includes harmful material according toone embodiment of the invention;

FIG. 12 is a flow diagram illustrating exemplary operations forredirecting requests directly to origin servers according to oneembodiment of the invention;

FIG. 13 is a flow diagram illustrating exemplary operations forvalidating whether a request should be subject to restriction afterdetermining that its IP address is listed on a restricted list accordingto one embodiment of the invention;

FIG. 14 is a flow diagram illustrating exemplary operations fortarpitting a visitor according to one embodiment of the invention;

FIG. 15 is a flow diagram illustrating exemplary operations forperforming response related actions according to one embodiment of theinvention;

FIG. 16 is a flow diagram illustrating exemplary operations performed bythe proxy server when responding to server offline errors according toone embodiment of the invention;

FIGS. 17A-B are flow diagrams illustrating exemplary operationsperformed by the proxy server for determining whether and how to modifythe content of a response according to one embodiment of the invention;

FIG. 18 is a flow diagram illustrating exemplary operations forobfuscating an email address according to one embodiment of theinvention;

FIG. 19 is a flow diagram illustrating exemplary operations forprocessing server side defined modification tokens according to oneembodiment of the invention;

FIG. 20 is a flow diagram illustrating exemplary operations for addingtrap email address(es) and/or trap form(s) to the content of a responseaccording to one embodiment of the invention;

FIG. 21 is a flow diagram illustrating exemplary operations for addingor changing advertisements to requested resources according to oneembodiment of the invention;

FIG. 22 is a flow diagram illustrating exemplary operations for acustomer of the service to input threat information about one or morevisitors according to one embodiment of the invention;

FIG. 23 is a block diagram illustrating an exemplary threat reportinginterface for customers according to one embodiment of the invention;

FIG. 24 is a block diagram illustrating an exemplary threat type formaccording to one embodiment of the invention;

FIG. 25 is a flow diagram illustrating exemplary operations for usingcustomer defined threat information to assign threat scores to visitorsaccording to one embodiment of the invention;

FIG. 26 is a flow diagram illustrating exemplary operations forcalculating a customer reputation score according to one embodiment ofthe invention; and

FIG. 27 is a block diagram illustrating an exemplary computer systemaccording to one embodiment of the invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth.However, it is understood that embodiments of the invention may bepracticed without these specific details. In other instances, well-knowncircuits, structures and techniques have not been shown in detail inorder not to obscure the understanding of this description. Those ofordinary skill in the art, with the included descriptions, will be ableto implement appropriate functionality without undue experimentation.

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to effect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other.

Methods and apparatuses for providing Internet-based proxy services(hereinafter “service”) is described. The service, which is availableover the Internet and does not require customers (e.g., owners of adomain and/or personnel working on behalf of domain owner to installhardware or software, allows customers (e.g., owners of a domain)) toprotect their network against Internet-based threats; empower fast,reliable, and robust performance from their network resources; assist inprotecting the Internet community by proactively stopping botnets,cleaning viruses, trojans, and worms; or any combination thereof. Unlikeother products that require installation of hardware or software, theservice described herein exists at the network level (and thus does notrequire customers to install hardware or software). In some embodiments,the service provides performance services for the customers. Forexample, the service can participate in a content delivery network (CDN)and dynamically cache customer's files closer to visitors. As usedherein, a visitor is an entity causing an instance of a client networkapplication (e.g., a web browser, an FTP (File Transfer Protocol)client, an SSH (Secure Shell) client, a Telnet client, etc.)implementing a network protocol to access content through a network(e.g., the Internet). A visitor can be a human user or a bot (a softwareapplication that automatically performs Internet related tasks).Examples of bots include search engines or other crawlers (e.g., emailharvesters, indexers, etc.).

In some embodiments, customers register for the service by changingtheir authoritative name server to an authoritative name server of theservice, and also changing the IP address(es) that resolve to theirorigin server(s) (which hosts content of their domain) to point to aproxy server of the service. In other embodiments, customers of theservice change individual DNS records to point to a proxy server (orpoint to other domain(s) that point to a proxy server of the service).For example, the customers may change their DNS records to point to aCNAME that corresponds with a proxy server of the service. Regardless ofthe embodiment, requests from visitors for actions to be performed onidentified resources of the customer's domain are received at the proxyserver.

The proxy server analyzes the requests and performs one or more requestrelated actions. For example, for each request, analyzing the requestincludes performing one or more of the following: determining whetherthe visitor making the request is allowed access to the requestedcontent; determining whether the visitor poses an Internet securitythreat (e.g., is a bot, is infected with a virus or other vulnerability,etc.); determines whether the request itself poses an Internet securitythreat (e.g., an SQL injection attack, etc.); determines whether therequest is malformed; determines the type and/or size of the requestedcontent; determines whether the origin server is offline; and determineswhether the requested content is available in cache. Based on theresults of the analyzing, the proxy server takes appropriate requestrelated actions. For example, the proxy server may respond to therequest locally (e.g., by blocking the request, displaying an indicationthat the visitor may be infected with a virus, worm, or othervulnerability, serving cached content, etc.) and may transmit therequest to the appropriate origin server for processing.

The responses from the origin servers may also pass through the proxyserver, which may analyze the response and perform one or more responserelated actions. For example, the proxy server may perform one or moreof the following when analyzing the response: determining whether theresponse poses an Internet security threat (e.g., whether the contentincludes a virus, worm, or other vulnerability); determining whether theresponse includes one or more elements that are to be excluded frombeing delivered to the visitor; determining whether to modify element(s)of the response; determining whether to obfuscate elements of theresponse (e.g., obfuscating an email address such that it will bedisplayed on the rendered page but obfuscated from the page source);determining whether to add content to the response; and determiningwhether to cache the contents. Based the results of the analyzing, theproxy server takes appropriate response related actions.

FIG. 1 illustrates an exemplary architecture of the service according toone embodiment of the invention. The domain owners 135A-L are customersof the service and register their respective domains for the service.For example, the authoritative name servers for each the domains of thedomain owners 135A-L are changed to the authoritative name server 142 ofthe service at operation 180. It should be understood that the backupauthoritative name servers serving the domains may also be changed to anauthoritative name server of the service. The zone file records for thedomains are also changed such that DNS resolution requests for thedomains owned by the domain owners 135A-L, which correspond with theorigin servers 130A-L respectively, resolve to the proxy server 120, atoperation 182. In one embodiment, customers (e.g., the domain owners135A-L or other entity (e.g., web administrators) on behalf of thedomain owners 135A-L) may use the service server 125 to change theirauthoritative name server to the authoritative name server 142 andchange their zone file to have their domain point to the service proxyserver (herein after “proxy server”) 120.

The service server 125, operated by the service, provides a set of toolsand interfaces for the domain owners 135A-L and is accessible over theInternet. For example, the service server 125, among other things,allows the domain owners 135A-L to register for the service, viewstatistics/logs of events, and report suspicious events. The serviceserver 125 includes tools to assist the domain owners 135A-L in changingtheir authoritative name servers and zone file record. It should beunderstood, however, that the domain owners 135A-L may change theirauthoritative name server and zone file without use of the serviceserver 125 (i.e., they may directly change the authoritative name serverand zone file).

The DNS system 140 is used to refer to the DNS system as a whole andincludes multiple DNS servers to resolve DNS requests. As illustrated,the DNS system 140 includes the authoritative name server 142, which isan authoritative name server for the service. Thus, the authoritativename server 142 is the authoritative name server for the domainscorresponding to the origin servers 130A-L. Accordingly, when the DNSsystem 140 resolves a request for a domain corresponding to one of theorigin servers 130A-L, the authoritative name server 142 provides theauthoritative answer. It should be understood that the DNS system 140includes more DNS servers (e.g., preferred domain servers, top-leveldomain name servers, other domain servers) than illustrated. It shouldalso be understood that there may be multiple authoritative web serversfor the service and they may be geographically distributed.

The client devices 110A-I are computing devices (e.g., laptops,workstations, smartphones, palm tops, mobile phones, tablets, gamingsystems, set-top boxes, etc.) that are capable of accessing networkresources (e.g., they include software such as web browsers that arecapable of accessing network resources). Users at the client devices110A-I request network resources (e.g., HTML pages, images, wordprocessing documents, PDF files, movie files, music files, or othercomputer files) through a client network application such as a webbrowser or other application (e.g., FTP client, SSH client, Telnetclient, etc.). The client devices 110A-I may be susceptible to beinginfected with viruses, worms, or other vulnerabilities and may be partof a botnet network.

The origin servers 130A-L are computing devices that serve networkresources (e.g., HTML pages, images, word processing documents, PDFfiles, movie files, music files, or other computer files). The originservers 130A-L respond to requests for network resources (e.g., from anHTTP request, FTP request, telnet request, etc.). Although notillustrated in FIG. 1, it should be understood that the networkresources of the origin servers 130A-L may be stored separately from thedevice that responds to the requests.

The proxy server 120 is a computing device that is situated between theclient devices 110A-I and the origin servers 130A-L and provides many ofthe features of the service. Certain network traffic passes through theproxy server 120 (traffic sent from the client devices 110A-I and/ortraffic sent from the origin servers 130A-L). Based on at least in parton this traffic, the proxy server 120 provides a set of one or moreservices for the benefit of the customers and/or users of the clientdevices 110A-I. For example, the proxy server 120 may provide one ormore of the following services: participation in a content deliverynetwork by providing cached files of the origin servers 130A-L (e.g.,through the cache 122); providing cached copies of files (if available)of the origin servers 130A-L during periods when they are offline (e.g.,through the cache 122); restricting access to the origin servers 130A-L(which may be based on a set of one or more factors such as thecharacteristics of the requester, the type of request, and the contentof the request); scanning the traffic (sent from a client device 110and/or sent from an origin servers 130) for vulnerabilities (e.g.,virus, worm, etc.) and acting accordingly (e.g., blocking the request,alerting the sender and/or receiver of the vulnerability, throttling theconnection to slow down the request, etc.); and modifying the content ofthe request and/or the reply (which may be based on a set of one or morefactors such as the content of the request, the content of the reply,and the characteristics of the requester).

While FIG. 1 illustrates a single proxy server 120, in some embodimentsthe service has multiple proxy servers that are geographicallydistributed. For example, in some embodiments, the service uses multiplepoint of presences (POPs). A POP is a collection of networking equipment(e.g., authoritative name servers and proxy servers) that aregeographically distributed to decrease the distance between requestingclient devices and content. The authoritative name servers have the sameanycast IP address and the proxy servers have the same anycast IPaddress. As a result, when a DNS request is made, the network transmitsthe DNS request to the closest authoritative name server. Thatauthoritative name server then responds with a proxy server within thatPOP. Accordingly, a visitor will be bound to that proxy server until thenext DNS resolution for the requested domain (according to the TTL (timeto live) value as provided by the authoritative name server). In someembodiments, instead of using an anycast mechanism, embodiments use ageographical load balancer to route traffic to the nearest POP.

In some embodiments, the proxy server 120 maintains a log of events. Forexample, the proxy server 120 logs each request that is received as wellas each response that is transmitted to a visitor client device. Inaddition, in embodiments where the proxy server 120 participates in aCDN, the proxy server 120 cases each request and whether there was acache hit (the requested resource is in the cache) or a cache miss (therequested object is not in the cache and the origin server was queried).Each proxy server in the service is assigned a unique identifier. Eachlog stores the server's unique identifier plus a timestamp plusadditional information about the event that generated the log (e.g., theIP address of the visitor client device initiating the request, theprevious server that relayed the request, the response, etc.).

Each proxy server maintains its own logs and reports events in the logsto the service server 125 over the network. While in one embodiment theevents are reported to the service server 125 in real time, in otherembodiments the events are reported to the service server 125differently (e.g., event logs are batched and transmitted to the serviceserver 125 at regular intervals or as the system resources allow, eventlogs are batched and downloaded from the service server 125 at regularintervals or as the system resources allow, or other possiblemechanisms).

The service server 125 maintains an event log data structure 126, whichstores the events of the proxy servers. The service server 125 sorts thelogs in order to retrieve multiple requests that constitute a singlesession by any particular visitor, even if the visitor's requests weremade to multiple proxy servers. Customers of the service can access theservice server 125 to view data reports such as such as a list of IPaddresses that have visited one of their website(s) protected by theservice, average time between page loads of particular IP address,whether a particular visitor downloads all the resources of a web pageor just the HTML, etc. The service server 125 can also report data suchas the percentage of visitors to a web site that are search engines, thepercentage that are humans, the percentage that were blocked, etc. Inone embodiment, the visitors that that are threats as displayed tocustomers can be sortable by the type of threat (e.g., email harvester,comment spammer, etc.), suspicious but not known to be a threat, and/orsortable by a known threat level.

While FIG. 1 illustrates multiple origin servers 130A-L coupled with theproxy server 120, in some embodiments the proxy server is coupled with asingle origin server. Moreover, in some embodiments, there are multipleproxy servers providing service for a particular domain.

The owner of the proxy server 120 is typically different than the ownersof the origin servers 130A-L. In addition, the proxy server 120 is nottypically part of the local network of the origin web servers 130A-L.For example, the proxy server 120 is outside of the local area networkof the origin web servers 130A-L and is typically not physicallyaccessible by owners/administrators of the origin servers 130A-L.

The validating domain server 180 a computing device that is used tovalidate whether a request should be subject to restriction if its IPaddress is listed on a restricted list. Since IP addresses may not bestatic (e.g., they may be assigned dynamically through DHCP, may changewho they are assigned to over time, and are subject to being hijacked orspoofed), it is possible for an IP address to be listed on a restrictedlist even though the visitor currently associated with that IP addresswas not responsible for the IP address being added to the restrictedlist. The validating domain server 180 determines whether the listing onthe restricted list is valid. For example, after the proxy server 120determines that an IP address of a request is listed on a restrictedlist (which will be described in greater detail later herein), the proxyserver 120 redirects the corresponding client device to the validatingdomain server 180. The validating domain server 180, which correspondswith a validating domain for the service, stores and reads cookies forthe validating domain in the global cookie database 185. Each cookieincludes an indication whether a client device should be allowedunrestricted access or be subject to its IP address being on arestricted list. For example, as will be described in greater detaillater herein, a human user of a client device may prove that he or sheis not a bot, which can be represented in their corresponding globalcookie.

The client devices 110A-I request DNS resolution when a domain name isused or requested by a local application and is not known (e.g., is notin a local DNS cache or the DNS record in its local cache has expired).Consider the following example, where a user of the client device 110Aenters the website example.com into a web browser of the device (theorigin server 130A serves the website example.com). If the client device110A does not know the IP address of example.com, (e.g., the cache onthe client device 110A does not have an entry for example.com or it hasexpired), the client device makes a DNS request 150 to the DNS system140 for the IP address for example.com. The domain owner of example.comhas changed its authoritative name server to the authoritative nameserver 142, and the DNS zone file has been changed so that the IPaddress returned by the authoritative name server 142 will be that ofthe proxy server 120. As such, the DNS system 140 performs a recursiveor iterative DNS process until the authoritative name server 142 returnsthe IP address for the proxy server 120 in the DNS response 152.

Sometime after the DNS resolution is complete and the client device 110Alearns the IP address that points to example.com (which is the IPaddress of the proxy server 120), the client device 110A makes therequest 154 (e.g., an HTTP GET request, an HTTP POST request, other HTTPrequest method, or other request for an action to be performed on anidentified resource belonging to an origin server), which is transmittedto the proxy server 120. The proxy server 120 analyzes the request atoperation 164 and determines a set of one or more request relatedactions to perform based on the results of the analyzing. In oneembodiment, for each request 154, the proxy server 120 performs one ormore of the following: determines whether the visitor making the requestis allowed access to the requested content; determines whether thevisitor poses an Internet security threat (e.g., is a bot, has a virus,has previously been identified as performing malicious activities (e.g.,email spamming, comment spamming, SQL injection attacker, participant ina denial of service attack, etc.), etc.); determines whether the requestitself poses an Internet security threat (e.g., an SQL injection attack,etc.); determines whether the request is malformed; determines the typeand/or size of the requested content; determines whether the originserver is offline; and determines whether the requested content isavailable in cache.

In some embodiments, the service maintains the threat database 124,which contains information that indicates whether the visitor poses athreat and whether the visitor is allowed access to the requestedcontent. The proxy server 120 accesses the threat database 124 throughthe query threat database operation 165. While in some embodiments thethreat database 124 is a central database common to multiple proxyservers, in other embodiments the threat database 124 is part of adistributed database system.

In some embodiments, the threat database 124 includes one or more of thefollowing: a global restricted IP address list that identifies IPaddresses that are not allowed to access content of any of the originweb servers protected by the service (e.g., the origin servers 130A-L);a local restricted IP address list for each of the origin servers 130A-Lthat identifies IP addresses that are not allowed access to that originserver (which may be based on location for the IP addresses); a localrestricted cookie list for each of the origin servers 130A-L thatidentifies cookies that are not allowed access to that origin server;global allow cookie list common to the origin servers 130A-L thatidentifies cookies that are allowed access to each of the origin servers130A-L; a local allow cookie list for each of the origin servers 130A-Lthat identifies cookies that are allowed access to that origin server;vulnerability signatures to detect viruses, worms, trojan horses, andother vulnerabilities; and visitor statistics. In some embodiments, thethreat database also includes the reason why a visitor is on arestricted list (e.g., email harvester, email and/or comment spammer,participated in denial of service attack, is infected, etc.).

The IP addresses on the global restricted IP address list are populatedby the service and may be added for numerous reasons. For example, IPaddresses of email address harvesters; IP addresses that have beendetected as belonging to comment or blog spammers; IP addresses thatcorrespond with SQL injection attacks or other web softwarevulnerability attacks; and IP addresses that have been detected asparticipating in denial of service attacks may be added to the globalrestricted IP list. Similarly, IP addresses may be added to a localrestricted IP address list of an origin server 130 after it determines,from a previous session, that the IP address has been recorded as apotential threat to that origin server 130 (e.g., email harvesting,comment or blog spamming, SQL injection attacks or other web softwarevulnerability attacks, denial of service attacks, etc.). The localrestricted IP address list may also be populated with IP addressclassifications that are not allowed access to the corresponding originserver. As an example of an IP address classification, a domain owner135 may restrict access to their origin server 130 to IP addresses of acertain location (e.g., country(ies)). In some embodiments, the globalrestricted IP address list and/or the local restricted IP address listindicates the reason that each IP address is included on the list (e.g.,email address harvester; blog or comment spammer; SQL injection attacksor other web software vulnerability attacks; DoS attacks; etc.).

Cookies may be added to the global allow cookie list after it isdetermined that the visitor does not pose a threat. For example, theservice may provide a mechanism for a user of a client device to verifythat he or she is a human user and not a bot. For example, the servicemay direct the user to respond to a CAPTCHA and record their input inresponse to a graphical image, audio recording, math problem, or otherchallenge-response test. If successfully passing the CAPTCHA, theservice may associate the cookie with a human user and add to the globalallow cookie list. Similarly, a domain owner may cause cookies to beadded to its local allow cookie list after determining, from a previoussession, that the visitor associated with the cookie is not a potentialthreat to the origin server. A domain owner may cause cookies to beadded to its local restricted cookie list after determining, from aprevious session, that the cookie has been recorded as a potentialthreat to the origin server (e.g., email harvesting, comment or blogspamming, SQL injection attacks or other web software vulnerabilityattacks, denial of service attacks, etc.).

In some embodiments, the request analyzing operation 164 includes theproxy server 120 determining whether the cookie of the request 154 (ifone is included in the request) or other header of the request 154 ismalformed. Malformed cookies or headers serve as an indication that therequest 154 is not from a human user and is a likely indication ofsuspicious activity. An example of a malformed header occurs if an HTTPGET request does not contain a URL and an HTTP version string, theheader is longer than the prescribed buffer size, the header containsdisallowed characters (e.g., non ASCII characters), the header containssignature(s) of known SQL injection attack(s), the header is too shortor incomplete, the header does not accurately reflect the content (e.g.,the content-length header may report a different length of content thanis actually in the body), the header may exclude the referrer or includea disallowed referrer, etc. The system can stop requests depending onthe content of the headers. For example, if the web administrator haschosen to block direct requests for images then the proxy server willnot return image content if the referrer does not include the webadministrator's website's domain.

In some embodiments, customers of the service can configure the servicesuch that the proxy server 120 will block requests depending on thecontent of the headers. By way of example, if a customer has configuredthe service to block direct requests for images then the proxy server120 will not return image content if the referrer does not include theprotected domain (or alternatively an image is returned that indicatesthat the direct loading of images has been blocked).

In some embodiments, the proxy server 120 redirects a client device 110directly to an origin server 130 based on the type of content requestedand/or the size of the requested content. For example, if the type ofthe content is not supported (e.g., it is a video file) and/or if thesize of the requested content is above a threshold, the proxy server 120may redirect the requesting client device directly to the origin serverso that the traffic passes directly between the client device and theorigin server. Thus, in these embodiments, the request analyzingoperation 164 includes the proxy server 120 determining the type and/orthe size of the requested content.

Based on the results of the analyzing request operation 164, the proxyserver 120 takes one or more appropriate request related actions.Examples of request related actions that may be performed by the proxyserver 120 include the proxy server 120 responding to the requestlocally by transmitting a response 162 (e.g., an HTTP response) to theclient device 110A, blocking the request 154 in addition to or in placeof the response 162, reducing the speed at which content can bedelivered to the client device 110A, and transmitting the request to theorigin server 130A on behalf of the client device 110A at operation 156.

In cases where the proxy server 120 locally transmits the response 162to the client device 110A (locally referring to transmitting theresponse 162 without the request being forwarded to the origin server130A), the response 162 may be different depending on different resultsof the analyzing request operation 164. Additionally, the response 162may be customized based on the characteristics of the requested websiteand/or the characteristics of the visitor.

As one example of the response 162, if the proxy server 120 determinesthat the visitor poses a likely Internet security threat by possiblybeing infected with a virus, worm, or other vulnerability and/orperforms malicious activities (e.g., as indicated by being listed on arestricted list, having an out of date system (e.g., a browser versionthat is known to have vulnerabilities)), the response 162 may include anindication that the visitor poses a potential threat (e.g., is likelyinfected with a virus, worm, or other vulnerability, has an out of datesystem, etc.) and may include instructions for the user to remedy thevulnerability (e.g., run an anti-virus software program, download andrun an anti-virus software program, download patches, download updatedsoftware version, etc.). In some embodiments, the proxy server 120determines that a visitor poses a likely Internet security threat byquerying the threat database 124 to determine if the IP address of thevisitor is included on one or more of the global restricted IP addresslist and/or the local restricted IP address list for the origin server130A, and/or to determine if the cookie of the visitor (if the request154 includes a cookie) is included on the local restricted cookie listfor the origin server 130A.

As another example, if the proxy server 120 determines that the visitorposes an Internet security threat because the visitor performs maliciousactivities and is likely an automated bot, the response 162 may includea block page indicating that the visitor has been blocked and mayinclude a mechanism for the visitor to dismiss the block page byverifying that a human user is using the client device. For example, animage, audio, or other CAPTCHA may be included in the response 162 toallow a human user to verify that he or she is not an automated bot. Ofcourse, it should be understood that in some embodiments the response162 may not include such a mechanism for the visitor to dismiss theblock page.

As another example, if the proxy server 120 determines that the requestitself poses a security threat (e.g., it contains malicious code) orincludes a malformed cookie or header, the response 162 may display anindication that the request cannot be completed and may indicate thereason (e.g., the request poses a security threat, the request includesa malformed cookie or header, etc.).

As another example of the response 162, if the proxy server 120determines that the request 154 should be redirected directly to theorigin server 130A (e.g., the type of the requested content is notsupported to flow through the proxy server 120 and/or if the size of therequested content is above a threshold), the response 162 indicates aredirection that points to an IP address of the origin server 130A(e.g., an HTTP response status code 301, 302, 303, 305, 307, or otherredirection status code that indicates a redirection to a subdomainwhose IP address points directly to the origin server 130A, or a pagewith a meta redirect and/or a script instructing the client networkapplication to load a page to the subdomain whose IP address pointsdirectly to the origin server 130A). After receiving a redirectionresponse, the client device 110A makes the redirected request 170directly requests to the origin server 130A. The origin server 130Adirectly responds to the request 170 with the direct response 172.

As another example of the response 162, if the proxy server 120determines that the requested content is available in the cache 122through the request and receive content operation 160, the response 162may include the requested content. The cache 122 may include a generalpurpose cache and a separate special purpose cache. The general purposecache is populated by human users and automated crawlers (e.g., searchengines or other crawlers parsing and indexing the websites) using theservice (e.g., requesting content from the origin servers 130A-L). Insome embodiments, the data in the general purpose cache is static data(e.g., images, videos, etc.) and does not include dynamic data (e.g.,HTML pages). In some embodiments, the proxy server 120 serves contentfrom the general purpose cache to human visitors and bots when possible.

The special purpose cache is populated by trusted searchengines/crawlers that request content through the service. A trustedsearch engine/crawler is one that only access publicly accessiblecontent (and thus does not access private content that requires ausername/password). In one embodiment, the service maintains a list ofpairs of IP addresses and user agents of trusted searchengines/crawlers. Unlike the general purpose cache, the special purposecache can include dynamic data (e.g., HTML pages) and optionally staticdata. Since trusted search engines/crawlers only access publiclyaccessible content, the data that is cached in the special purpose cachewill not include private information. By way of specific example, thetrusted search engine/crawlers may access publicly accessible pages of abanking website but cannot access pages of user accounts at the bankingwebsite (which would require a logon). Accordingly, the special purposecache will not include the pages of user accounts at the bankingwebsite. In some embodiments, the proxy server 120 serves content fromthe special purpose cache to human visitors only when the origin serveris offline, and serves content from the special purpose cache to botvisitors (e.g., search engines, crawlers, etc.) when possible.

In some embodiments, the proxy server 120 does not cache certain filesin the cache 122 that are not supported by the service or those filesthat have a size above a certain threshold. While in some embodimentsthe general purpose cache and the special purpose cache are separate andlocated in separate databases, in other embodiments they are part of thesame database and/or located on the same computing device.

Prior to transmitting the content to the client device 110A, the proxyserver 120 may analyze the content and modify the content. For example,in some embodiments, if the content is an HTML page that includes anembedded email address, the proxy server 120 obfuscates the emailaddress such that it will be displayed on the rendered page but will notbe readable from the page source, thus preventing the email address frombeing harvested by an email harvesting program. As another example, thedomain owner 135A may define rules that certain elements of the contentare to be excluded from the response 162 depending on one or morecharacteristics of the visitor. If the rule for an object is triggered,the proxy server 120 removes that content from the response 162.

In some embodiments, the proxy server 120 reduces the speed at whichcontent can be delivered to a client device 110 responsive todetermining that the visitor and/or the request 154 is a potentialthreat. For example, the proxy server 120 turns down the number of bytesper second that can be delivered through it for the connection.

FIG. 2 is a block diagram illustrating an exemplary request module ofthe proxy server 120 that processes requests according to one embodimentof the invention. The request module 210 includes the request analyzer220, the request threat manager 225, the redirection module 230, and thecache response module 235. The request analyzer 220 receives andanalyzes the request 154. The request analyzer parses the request 154and performs one or more of the following: determines the destination ofthe request (by examining the header of the request); determines therequested content (by examining the header of the request); determineswhether the visitor poses an Internet security threat; determineswhether the request itself poses an Internet security threat; determineswhether the request is malformed; determines the type and/or size of therequested content; determines whether the origin server is offline; anddetermines whether the requested content is available in cache and isappropriate to transmit to the visitor.

In one embodiment, the request analyzer 220 determines whether a visitorposes an Internet security threat by querying the threat database 124 todetermine whether the visitor is on a restricted list (e.g., the globalrestricted IP address list, the local restricted IP address list, and/orthe local restricted cookie list) and/or based on visitorcharacteristics (e.g., to determine whether the visitor is a human useror a bot). In one embodiment, the request analyzer 220 creates a visitorfingerprint based on a set of one or more visitor characteristicsresponsive to a visitor making a request 154. For example, the visitorfingerprint is created based on one or more of the following: whetherthe client network application loads images; whether the client networkapplication executes JavaScript; the type of network application (e.g.,browser name and version); the operating system running the clientnetwork application; the fonts installed on the client networkapplication; the languages supported by the client network application;whether the client network application supports plugins; whether theclient network application stores cookies; whether the client networkapplication responds from the same IP address for various protocolrequests, etc.

The request analyzer 220 can determine much of the information for thevisitor fingerprint based on the information in the header of therequest 154. For example, the IP address of the visitor, the particularplugins and extensions that the client network application supports(e.g., Flash, PDF, etc.), the fonts installed on the client networkapplication, the User-Agent of the client network application, thescreen size, the content types that are accepted, the character setsthat are accepted, whether compressed content is accepted, the acceptedlanguages, the time zone of the client network application, and whetherthe client network application has a cookie for the visited site, can bedetermined from the information in the header of the request 154.

The proxy server 120 can also generate data by injecting input into theresponse and/or analyzing the logs generated by the visit. For example,the proxy server 120 can insert a script into the HTML page returned tothe visitor. The script, if executed, makes a call to a URL monitored bythe proxy server 120 (or other device of the service). The serviceserver 125 can monitor the event log database for requests to that URLfrom the IP address and/or cookie of the original visitor. The serviceserver 125 can correlate the original request with the request made viathe script. If the script call is not made, the service server 125 caninfer that the client network application did not execute the script. Insome embodiments, the service server 125 can use a specially constructedURL that corresponds to a particular HTML request in order to furtherassociate the original request with the script-generated request down toa particular page.

In addition, the service server 125 can examine the loading of otherresources on a page returned to a visitor in order to determine thecharacteristics of the client network application. For example, if a webpage is made up of an HTML document and a number of images, the serviceserver 125 can examine the event logs for the visitor in order todetermine whether the images were loaded when the HTML page was loaded.If a particular visitor repeatedly does not load images across multipleweb pages, the service server 125 can infer that the visitor has imageloading blocked or turned off. In certain circumstances, this may be anindication that the visitor is a bot, and not a human.

In some embodiments, the request analyzer 220 uses the visitorfingerprint to make a likely determination of whether the visitor is ahuman user or is a bot. For example, there may be inconsistencies in thecharacteristics of a visitor that lead to the determination that thevisitor is a bot. For example, if the request is from a browser that isknown to use a certain font but that font as indicated by thecharacteristics is not installed, then it is likely that at least partof the request has been forged and it is likely not a human user withlegitimate intentions. According to one embodiment, the moreinconsistencies between characteristics the more likelihood that thevisitor is not a human user with legitimate intentions and thus poses athreat.

The request analyzer 220 determines whether the request itself poses anInternet security threat by analyzing the content of the request. Forexample, if the cookie or header is malformed then the request may posea threat (a malformed cookie or header is an indication that the requestis not from a human user and is a likely indication of suspiciousactivity). The request analyzer 220 may determine whether the URL of therequest is malformed and/or contains a known threat signature. If therequest 154 is a POST request, the request analyzer 220 analyzes thecontents that are attempted to be posted for potential threats (e.g.,against known threat signatures).

In some embodiments, the request analyzer 220 determines whether therequested content is available in the cache 122 and is appropriate totransmit to the visitor (based on the visitor characteristics). Forexample, if the requested content is for relatively static data (e.g.,images, videos, etc.) whose content is not likely to change often, andis available in the cache 122, then the cached version can be returnedto the visitor. As another example, if the requested content is forrelatively dynamic data (e.g., an HTML page) whose content is likely tochange rather frequently, then the cached version may not be appropriateto be sent to human users (unless the server is determined to beoffline).

If the request analyzer 220 determines that the visitor is a threatand/or that the request 154 is itself a threat, then it calls therequest threat manager 225. If the request analyzer 220 determines thatthe file type of the requested content and/or the size of the requestedcontent is not supported to be passed through the proxy server 120, therequest analyzer 220 calls the redirection module 230 (in someembodiments, as a prerequisite for calling the request threat manager225, the request analyzer 220 first determines that the visitor and/orthe request is not a threat). If the request analyzer determines thatthe requested content is available in the cache 122 and is appropriateto transmit to the visitor, then the request analyzer 220 calls thecache response module 235 (in some embodiments, a prerequisite forcalling the cache response module 235 is determining that the visitorand/or the request is not a threat). If the request analyzer determinesthat the visitor and/or the request is not a threat, the requestanalyzer may cause the request 156 to be transmitted to the appropriateorigin server (in some embodiments, the request analyzer 220 alsodetermines that the requested content is not available in the cache 122(or is not appropriate to transmit to the visitor) and/or the requestshould not be redirected directly to the origin server).

The request threat manager 225 manages threats differently in differentembodiments. For example, the request threat manager 225 may block therequest 154 without a response 162 being transmitted to the visitor,block the request 154 and generate a block page for the response 162that indicates that the request was blocked (the block page may includea mechanism for the visitor to dismiss the block page), cause a responseto redirect the request to the validating domain server 180, anddecreases the speed at which content can be delivered to the visitor. Inone embodiment, the request threat manager 225 treats threatsdifferently based on a relative threat level. For example, if a visitoris located on multiple restricted lists such as a restricted IP addresslist (the global and/or local restricted IP address list) and the localrestricted cookie list, then the threat level is relatively high and therequest threat manager 225 may block the request without providing amechanism for the visitor to dismiss the block page. As another example,if the visitor is not located on a restricted list but whosecharacteristics suggest that the visitor is a bot, then the threat levelis medium and the request threat manager 225 may block the request andprovide a mechanism for the visitor to dismiss the block page by provingthat they are a human user. In all cases, the request threat manager 225logs the activities of visitors interacting with a block page, which mayaffect the treatment of the visitors for future request (e.g., a visitorsuccessfully completing a CAPTCHA is less likely to be a threat and canbe labeled as such compared with a visitor failing one or more CAPTCHAswhich indicates the visitor is more likely to be a threat and can belabeled as such).

The redirection module 230 causes the response 162 to include aredirection status code to the origin server directly (e.g., an HTTPresponse status code 301, 302, 303, 305, 307, or other redirectionstatus code that indicates a redirection to a subdomain whose IP addresspoints directly to the correct origin server, or a page with a metaredirect and/or a script instructing the client network application toload a page of the domain whose IP address points directly to thecorrect origin server).

The request threat manager 225, redirection module 230, and the cacheresponse module 235 each call the response module 250 to generate anappropriate response to the visitor.

In some embodiments, the request 156 (e.g., an HTTP GET request, an HTTPPOST request, other HTTP request method, or other request for an actionto be performed on an identified resource belonging to an origin server)is transmitted by the proxy server 120 to the origin server 130A onbehalf of the client device 110A. In some embodiments, prior totransmitting the request 156, the proxy server 120 determines one ormore of the following: the visitor is allowed access to the content ofthe origin server 130A; the visitor does not pose an Internet securitythreat; the request 154 does not pose an Internet security threat; therequest 154 is not malformed; the requested content is of a type and/orsize not supported by the cache 122; and the requested content is not inthe cache 122.

The request 156 transmitted by the proxy server 120 to the origin server130A on behalf of the client device 110A may be substantially similar tothe original request 154 or it may be modified by the proxy server 120.For example, in some embodiments, the proxy server 120 removes contentfrom the request if it determines that the content is a security threatto the origin server 120 while leaving the content that is not asecurity threat to be transmitted (e.g., if the request is an HTTP POSTand the contents attempted to be posted contain a possible threat to theorigin server). In other embodiments, the proxy server 120 modifies thecontent of the request to make the request less likely to harm to theorigin server 120. For example, the proxy server 120 may escapedangerous characters. As an example, SQL injection attacks often includea quotation mark in an attempt to break out of the SQL on the proxyserver 120. To prevent this type of attack, the proxy server 120 can addan escape character (e.g., a “V”) before the quotation in the request.

The origin servers 130A-L respond to the request 156 as if the requestwas being transmitted from a client device directly. The response 158(e.g., an HTTP response) may include the requested content, an errorcode indicating that the content cannot be found (e.g., an HTTP responsestatus code 404 error), an error code indicating an problem with theorigin server (e.g., an HTTP response status code 5XX error) or otherresponse code.

After receiving the response 158, the proxy server 120 analyzes theresponse (at the analyzing response operation 166) and determines a setof one or more response related actions to perform based on the resultsof the analyzing response operation 166. The analyzing responseoperation 166 includes the proxy server 120 performing one or more ofthe following: determining the status of the response (e.g., whether itindicates an error code); determining whether the header of the responseis malformed; determining whether the response poses an Internetsecurity threat (e.g., whether the requested resource includes a virus,worm, or other vulnerability); determining whether the requestedresource includes one or more elements that are to be excluded frombeing delivered to the visitor; determining whether to modify element(s)of the requested resource; determining whether to obfuscate elements ofthe requested resource (e.g., obfuscating an email address such that itwill be displayed on the rendered page but obfuscated from the pagesource); determining whether to add content to the requested resource;and determining whether to cache the contents of the requested resource.Based on the results of the analyzing response operation 166, the proxyserver 120 performs one or more appropriate response related actions.

In some situations, the response 158 may include elements that are to beexcluded and/or obfuscated from the response 162 based on one or morecharacteristics of the visitor. For example, in some embodiments, thedomain owners 135A-L may each define modification rules that defines howto modify elements of the content prior to transmitting the response162, which are triggered by characteristics of the visitor. By way ofexample, a modification rule could be defined to modify elements withincontent for IP addresses of a certain location (e.g., a certaincountry). In some embodiments, the service server 125 provides a rulecreation interface that allows the domain owners to establishmodification rules. If a visitor triggers a modification rule to excludean element from the response, the proxy server 120 removes that elementfrom the response 162. If a visitor triggers a modification rule toobfuscate an element from the response, the proxy server 120 obfuscatesthat element by replacing the element with a script that, when executed(e.g., when the page loads on the client network application), generatesthe underlying data on the rendered page yet is unreadable in the sourceof the page.

In some embodiments, the proxy server 120 automatically modifiesportions of the requested content prior to transmitting the response162. For example, the proxy server 120 modifies known-sensitive contentof the requested content (e.g., phone numbers, email addresses, instantmessenger IDs, street addresses, links to other websites, birthdates,social security numbers, IP addresses, credit card numbers, accountusernames, etc.) in such a way that it cannot be parsed or indexedautomatically. For example, the proxy server 120 replaces the structureddata with an obfuscation script, which when executed (e.g., upon thepage loading in the client network application), generates the data suchthat it will be displayed on the rendered page but will not be readableby an automatic parser or indexer. By way of specific example, in someembodiments the proxy server 120 automatically obfuscates emailaddresses embedded in the response such that the email address will bedisplayed on the rendered page but will not be readable from the sourceof the page, thus preventing the email address from being harvested byan email harvesting program. For example, the proxy server 120 replacesthe email address with an obfuscation script, which when executed (e.g.,upon the page loading), generates the email address to be displayed onthe rendered page. While in some embodiments the proxy server 120obfuscates each email address detected in the requested content, inother embodiments the proxy server 120 scrambles the detected emailaddresses in the requested content only for certain visitors (e.g.,known search engines and/or known crawlers). An exemplary mechanism ofobfuscating an email address will be described with reference to FIG.18.

In some embodiments, customers may customize which elements should beautomatically obfuscated. For example, the service server 125 mayprovide an interface to allow the customers to select which elements(e.g., phone numbers, email addresses, instant messenger IDs, streetaddresses, links to other websites, birthdates, social security numbers,IP addresses, credit card numbers, account usernames, etc), if any,should be automatically obfuscated by the proxy server 120.

As another example of modifying the response, in some embodiments theproxy server 120 modifies resource unavailable errors (e.g., HTTPresponse status code 404 errors) with customized content. In someembodiments, the custom error page is customized based on one or more ofthe following: the location (e.g., country) of the visitor who triggeredthe error; the location of the origin server; the language of thevisitor who triggered the error; any cached content on the website wherethe error occurred, including the page that may have been cached beforethe error occurred or other pages on the site that give an overallcontext to the site generally (e.g., the type of site (e.g., sports,news, weather, entertainment, etc.)); a list of links or terms providedby the customer; the list of the most accessed pages elsewhere on thewebsite determined by other visitors; any terms that can be parsed fromthe request URL or POST; and any terms that can be parsed from thereferrer URL. Based on one or more of these factors, the error page mayinclude links to other pages on the web site or on other websites thatwould be of interest to the visitor. In some embodiments, the links aresponsored to advertisers looking to target individuals requestingparticular content.

In some embodiments, if the originally requested content has movedlocations (e.g., it is now available at a different URL), the customizederror page includes a link to a new location of the originally requestedcontent or automatically redirects the visitor to the new location ofthe originally requested content. In some embodiments, the proxy serverdetermines that requested content has moved locations by comparing ahash of the original content with hashes of other content from the sameorigin server when an error message occurs.

In some embodiments, the proxy server 120 adds content to the requestedresource in the response 162. For example, in some embodiments the proxyserver 120 adds a trap email address and/or a trap form to the response162. A trap email address is an email address that is not used for anyreal email and is unique to a particular IP address and session (thusthe email address will not be known or valid to different sessionsand/or computing devices). A trap form is a form to submit comments orother information (using a POST method or a GET method with variablesincluded in the URL) that is not used for any real comments. Similar tothe trap email address, the trap form will be unique to a particular IPaddress and session. The trap email address(es) and/or trap form(s) thatare added to the requested resource will not be displayed by the clientdevices 110A-I but are able to be harvested by an email harvestingprogram and used to submit data by an automated bot. Thus, the trapemail address(es) and/or trap form(s) are used to determine whethervisitors are human users or bots. The proxy server 120 (or othernetworking device of the service) monitors the email account thatcorresponds to the trap email address for emails. Since the trap emailaddress is not published and is not known by other computing devices,receipt of email at the account corresponding to that trap email addressis an indication that the IP address associated with the request 154belongs to an email harvesting program. In such a case, that IP addressmay be added to the threat database 124 (e.g., in the global restrictedIP address list). In addition, receipt of data through use of a trapform is an indication that the visitor associated with the request 154is a bot.

As another example of adding content to a requested resource, in someembodiments the proxy server 120 adds and/or changes advertisements tothe response 162. For example, in one embodiment of the invention, theproxy server 120 scans for advertisements in the content and replacesone or more of those advertisements with different advertisements. Theproxy server 120 performs one or more of the following when replacingadvertisements: determining the location, size, and position ofadvertisement(s) on a page; determining whether it is appropriate toreplace the advertisement(s); and replacing the advertisement(s) eitherdirectly or through a reference to an external resource. Theadvertisements may be modified in such a way that advertising blockers(software or features of browsers that prevent advertisements from beingdisplayed on web pages) are prevented from blocking the advertisements.

As another example of adding content to a requested resource, in someembodiments the proxy server 120 automatically adds scripts to pages(e.g., scripts that track users (e.g., the links they select, theduration of page visit, when they exit a page), scripts that trackperformance of a particular page (e.g., tracking page load times byincluding scripts in multiple places on the page), scripts that add ormodify content on the page (e.g., add affiliate codes to existing links,add links dynamically to content for particular keywords, add contentloaded from third party resources)). By way of specific example, a pageloading statistical script can be used to measure page load times andcan be inserted in one or more locations on the HTML page (e.g.,immediately after the <body> tag, after every X number of bytes, and/orimmediately before the </body> tag). The script may help customersmeasure page load time by causing the client network application to loadan image at a certain time during the page loading process. It should beunderstood that by the proxy server 120 automatically adding thesescripts to the pages avoids the customers from manually adding thesetypes of scripts. For example, in some embodiments, the customers canselect an option presented by service (e.g., through a customizationportal on the service server 125) for the proxy servers to automaticallyadd one or more of these scripts (e.g., page load times, etc.) to thepages that pass through the proxy or served from the cache.

In some embodiments, customers may choose (e.g., through a customizationportal on the service server 125) to add statistics scripts to only acertain percentage of resources. By way of example, a customer mayconfigure the service to include statistics scripts for only certaindemographics (e.g., operating system type, client network applicationtype, country of origin, time of day, number of times they havepreviously visited the site, etc.) and/or only for a certain percentageof visitors (the percentage being definable by the customer and/or theservice).

In some embodiments, the proxy server 120 caches the content received inthe response 158 to the cache database 122. In some embodiments, theproxy server 120 caches content that is supported and/or has a size thatis below a threshold. The proxy server 120 stores an indication (e.g., aflag) if the size of the content is above the threshold such that thenext request for that content will be redirected directly to the originserver. In embodiments where the response 158 is modified, in someembodiments the modified response is cached while in other embodimentsthe unmodified original response is cached.

FIG. 3 is a block diagram illustrating an exemplary response module 250according to one embodiment. The response module 250 includes theresponse and resource analyzer 255, the response threat manager 260, theserver error module 265, the unavailable resource module 270, theresource modification module 275, and the response forming andtransmission module 280. The response threat manager 260, the servererror module 265, the unavailable resource module 270, and the resourcemodification module 275 each call the response forming and transmissionmodule 280 to form and transmit responses 162 to the client devices.

The response and resource analyzer 255 receives and analyzes responses158 from origin servers and analyzes the content included in responses158 from origin servers or included from the cache 122. In someembodiments, when a response 158 is received, the response and resourceanalyzer 255 analyzes the response as described in the analyzingresponse operation 166. In some embodiments, the response and resourceanalyzer 255 also analyzes requested resources obtained from the cache122 to determine one or more of the following; whether the requestedresource includes one or more elements that are to be excluded frombeing delivered to the visitor; whether to modify element(s) of therequested resource; whether to obfuscate elements of the requestedresource (e.g., obfuscating an email address); and whether to addresource to the requested resource. If the response and resourceanalyzer 255 determines that the response 158 includes resources to becached, the cached version of the file will be cached in the cache 122.

If the response and resource analyzer 255 determines that there is athreat in the response or the response is malformed, the response andresource analyzer 255 calls the response threat manager 260. It shouldbe understood that the response may include a threat to the visitorand/or a threat to the proxy server 120. The response threat manager 260may remove the threat (e.g., remove the threatening content beforereturning the requested resource) or block the response (and/or alertthe visitor and/or customer that the response has been blocked).

If the response and resource analyzer 255 determines that the responseincludes a server error (e.g., the HTTP status code is a 4XX error, a5XX error, a timeout, a failure of DNS resolution, or content returnsthat indicate that the server is offline (e.g., database error or otherknown error pages)), then the server error module 265 is called. Theserver error module 265 causes the response 162 to include a cached copyof the requested resource (if available). If a cached copy is notavailable, the server error module 265 includes in the response 162 theerror message. In some embodiments, the server error module 265 adds ascript to the cached content (e.g., if the content is an HTML page) thatautomatically requests the proxy server 120 to ping the origin serverperiodically to determine whether the origin server is online. Theserver error module 265 may also set an offline browsing cookie for thevisitor such that if a subsequent request is received from the visitorwith the offline browsing cookie, the cached version of the requestedcontent will be served instead of querying the origin server.

If the response and resource analyzer 255 determines that the responseincludes a resource unavailable error (e.g., an HTTP status code 404error), then the unavailable resource module 270 is called. In oneembodiment, the unavailable resource module 270 modifies the error withcustomized content based on the requested content. For example, based onthe URL of the requested content, the response and resource analyzer 255modifies the response with suggestions of alternate pages.

If the response and resource analyzer 255 determines that the requestedresource includes elements that are to be modified (e.g., excluded,obfuscated, and/or added), then the resource modification module 275 iscalled. In some embodiments, the response and resource analyzer 255reads modification rules (which may be defined by domain owners) thatdefine how to modify elements of the requested and identifies thoseelements that should be modified (the visitors characteristics triggerthe rules). The resource modification module 275 modifies the resourceappropriately (e.g., removes the element or obfuscates the element). Insome embodiments, the resource modification module 275 adds content tothe requested resource. For example, the resource modification module275 may add trap email address(es) and/or trap form(s) to the requestedresource. As another example, the resource modification module 275 mayadd (or replace) advertisement(s) to the requested resource.

Registering for Service

FIGS. 4A-B are flow diagrams illustrating exemplary operations for acustomer to use the service server 125 to register for service accordingto one embodiment. The operations of FIGS. 4A-B will be described withreference to the service server 125; however it should be understoodthat the operations of FIG. 4A-B can be performed by embodiments otherthan those discussed with reference to the service server 125 and theservice server 125 can perform operations different than those discussedwith reference to the operations of FIG. 4A-B. In addition, theoperations of FIG. 4A-B will be described with reference to the domainowner 135A, which owns the domain example.com and is hosted by theorigin server 130A.

At block 410, the service server 125 receives the name of the domain(e.g., example.com) from the domain owner 135A. For example, withreference to FIG. 7A, the service server 125 provides the domain inputform 710 to allow the domain owner 135A to input their domain (e.g.,example.com) into the domain field 715. The domain owner 135A submitsthe domain information by selecting the submit button. Flow moves fromblock 410 to block 415.

At block 415, the service server 125 queries the global DNS system todetermine the authoritative name servers and domain name registrar forthe domain (e.g., example.com). Flow then moves to block 420, where theservice server 420 determines whether the current information in the DNSzone file for the domain is capable of being retrieved by the serviceserver 420 in order to avoid the domain owner 135A from inputting theinformation. For example, some DNS providers may provide an API(Application Programming Interface) that can be used by the serviceserver 420 to query for the information in the DNS zone file for thedomain. The list of DNS providers that provide such an API andinformation of how to use the API is stored by the service server 420.As another example, the service server 420 may simulate a human userlogging into the DNS provider's website to determine the information inthe DNS zone file. In such a case, the service server 420 accesses a mapof the DNS provider's website that has been pre-recorded by an operatorof the service and stored by the service server 420. The map includesthe web page on which the user login information is entered, theparticular fields into which the login information is entered, the pageor pages on which the zone information is displayed, the structure ofthose pages, and any links or URLs to request additional pieces of thezone file from the DNS provider. If the DNS zone file is capable ofbeing retrieved, then flow moves to block 425, otherwise flow moves toblock 435.

At block 425, the service server 125 receives login information (e.g.,username and password) to the DNS provider's website from the domainowner 135A. For example, with reference to FIG. 7B, the service server125 provides the DNS provider login information input form 710 to allowthe domain owner 135A to input their username and password for the DNSprovider 715 into the username field 720 and password field 725respectively. The domain owner 135A submits the login information to theservice server 125 by selecting the submit button. Flow moves from block425 to block 430.

At block 430, the service server 125 logs into the DNS provider websiteusing the login information and retrieves the information from the DNSzone file record for the domain. For example, if the DNS providerprovides an API for querying the information in the DNS zone file forthe domain, the service server 125 uses that API to query for the zonefile information. If there is not such an API, the registrations server125 queries the DNS provider via a service server-controlled agent(e.g., using HTTP or HTTPS protocols). For example, the service server125 may request the login page, enter any required login information,submit the login page, request one or more pages where the zone file isdisplayed, store the response from those pages, scan the pages based onthe predefined map to retrieve the zone information, and logout of theDNS provider. Flow moves from block 430 to block 440.

Referring back to block 435 (the information in the zone file is notcapable of being retrieved by the service server 125), the serviceserver 125 prompts the domain owner 135A to enter the information forthe DNS zone file record for the domain. For example, FIG. 5 illustratesan exemplary interface provided by the service server 125 to allowdomain owners to enter the information for the DNS zone file records. Asillustrated in FIG. 5, the interface 510 allows domain owners toindicate for each record a resource record type 515 (e.g., A, CNAME, NS,MX, LOC, etc.), a name 520, resource record type specific data 525, anda time-to-live (TTL) value 530. Flow moves from block 435 to block 440.The service server 125 may also provide a tool to assist the domainowner 135A in manually entering in the information to prevent mistakes.

FIG. 6 is a flow diagram illustrating exemplary operations performed bythe service server 125 to assist domain owners in manually entering DNSzone file information according to one embodiment. At block 610, theservice server 125 receives a keystroke input from the domain owner 135Afor one of the resource record type fields. For example, with referenceto FIG. 5, the domain owner 135A enters at least one keystroke in one ofthe fields 515, 520 and 525. Next, at block 615, the service server 125queries the global DNS system for the keystroke input to determine ifthere is a matching record. Flow then moves to block 620, where theservice server 125 determines whether the global DNS system indicatesthat there is no record for the queried entry. If there is no record,flow then moves back to block 610 where the domain owner 135A may entermore keystroke input. If there is at least one matching record, thenflow moves to block 625 where the service server populates the fieldwith one or more suggestions. Flow then moves to block 630, where if theservice server 125 receives another keystroke from the domain owner135A, then flow moves to block 635 where the suggestion(s) are clearedand flow moves back to 610. If more keystrokes are not entered by thedomain owner 135A, then flow moves to block 635 where the service server125 waits for the domain owner 135A to select one of the suggests or addmore keystrokes.

Referring back to FIG. 4A, at block 440, the service server 125 displaysthe zone file information to the domain owner 135A to allow the domainowner 135A to confirm its accuracy. The domain owner 135A may also editthe information if it is not accurate. Flow then moves to block 445where the service server 125 receives from the domain owner 135Adesignation of which records in the zone file are to be protected by theservice. For example, the domain owner 135A indicates at least that theaddress record (e.g., record type A or AAAA) of the domain (e.g.,example.com) is protected by the service. Flow moves from block 445 toblock 450.

At block 450, the service server 125 modifies the DNS zone record(s)designated by the domain owner 135A and the DNS authoritative nameservers for the domain to that of the service. For example, the addresspointing to the resource record type A (or AAAA) of the domain (e.g.,example.com) is changed to an IP address of a proxy server such as theproxy server 120, and the authoritative name servers are changed toauthoritative name servers of the service (e.g., including theauthoritative name server 142). The proxy server 120 may be one ofmultiple proxy servers in the service. The service server 125 may chooseone of the proxy servers in a number of ways (e.g., based on currentand/or expected load, based on location, round robin, etc.). Flow movesfrom block 450 to block 455.

At block 455, the service server 125 determines whether it supports anautomatic setup procedure to change the authoritative name servers atthe domain name registrar for the domain. For example, some domain nameregistrars may provide an API that can be used by the service server tochange the authoritative name servers for the domain. The list of domainname registrars that provide such an API and information of how to usethe API is stored by the service server 420. As another example, theservice server 420 may simulate a human user logging into the domainname registrar's website to change the authoritative name servers forthe domain. In such a case, the service server 420 accesses a map of thedomain name registrar's website that has been pre-recorded by anoperator of the service and stored by the service server 420. The mapincludes the login page, any fields where the login information isentered, the path to the page on which the authoritative name serversare changed, the fields that must be updated for those authoritativename servers to be changed, and any interface provided to delete nameservers. If the service server supports automatic changing of theauthoritative name servers at the domain name registrar for the domain,the flow moves to block 460; otherwise flow moves to block 465.

At block 460, the service server 125 receives login information (e.g.,username and password) to the domain name registrar's website from thedomain owner 135A. For example, with reference to FIG. 7C, the serviceserver 125 provides the domain name registrar login information inputform 710 to allow the domain owner 135A to input their username andpassword for the domain name registrar 715 into the username field 720and password field 725 respectively. The domain owner 135A submits thelogin information to the service server 125 by selecting the submitbutton. Flow moves from block 425 to block 430.

At block 430, the service server 125 logs into the registrar's websiteand updates the authoritative name servers to that of the service. Flowthen moves to block 475 where the service server 125 initiates a test tocheck to determine whether the authoritative name servers have beensuccessfully changed. For example, the service server queries the globalDNS system (e.g., with a dig operation, whois operation, etc.) for thedomain to confirm that the authoritative name servers have beensuccessfully changed. It should be understood that it may take someamount of time for the change of the authoritative name server topropagate throughout the global DNS system.

Request Related Actions

FIG. 8 is a flow diagram illustrating exemplary operations performed bya proxy server according to one embodiment. The operations of FIG. 8will be described with reference to the client device 110A and theorigin server 130A, which corresponds with the domain example.com. Atblock 810, the proxy server 120 receives the request 154 from the clientdevice 110A for content at the domain example.com. Flow moves from block810 to block 812 where the proxy server 120 analyzes the request (e.g.,similar to the analyzing request operation 164). Flow moves from block812 to block 815, where the proxy server 120 determines whether therequest and/or visitor is a threat.

FIG. 9 is a flow diagram illustrating exemplary operations to determinewhether a request and/or a visitor is an Internet security threataccording to one embodiment. The operations of FIG. 9 will be describedwith reference to the exemplary embodiment of FIG. 2. However, it shouldbe understood that the operations of FIG. 9 can be performed byembodiments of the invention other than those discussed with referenceto FIG. 2, and the embodiments discussed with reference to FIG. 2 canperform operations different than those discussed with reference to FIG.9.

At block 910, the proxy server 120 determines whether the request 154includes a valid customer bypass cookie. A customer bypass cookie allowscustomers to bypass any threat checking performed by the proxy server120. In one embodiment, responsive to a customer logging into theservice (e.g., through the service server 125), the service server 125opens an iFrame or other object (e.g., IMG tag, CSS (Cascading StyleSheets), etc.) that makes a request to the customer's origin server. Theproxy server 120 receives this request and returns a result with thecustomer bypass cookie set, which includes a unique code that identifiesthe client network application as belonging to the customer. Thecustomer bypass cookie can be stored in a database (or other datastructure) and associated with the domain, or it could be a hash of thedomain and a salt value. In the former case, if a customer bypass cookieis present in the request, the proxy server 120 accesses the database todetermine if the customer bypass cookie matches the correct value. Ifso, the customer bypass cookie is valid. In the latter case, if acustomer bypass cookie is included in the request, the proxy server 120hashes the requested domain plus the secret salt and compares it to thevalue of the customer bypass cookie. If they match, the customer bypasscookie is valid. If the request includes a valid customer bypass cookie,then flow moves to block 822 of FIG. 8 and the threat checking iseffectively bypassed. If the request does not include a valid customerbypass cookie, then flow moves to block 920.

At block 920, the proxy server 120 determines whether the IP address ofthe request 154 is on a global restricted IP address list. For example,the proxy server 120 queries the threat database 124 with the IP addressof the request 154 to determine whether it is on the global restrictedIP address list. If the IP address is on the global restricted IPaddress list, then flow moves to block 940; otherwise flow moves toblock 925.

At block 925, the proxy server 120 determines whether the IP address ofthe request is on the local restricted IP address list for the requesteddomain. For example, the proxy server 120 queries the threat database124 with the IP address of the request 154 to determine whether it is onthe local restricted IP address list. If the IP address of the request154 is on the local restricted IP address list, then flow moves to block940; otherwise flow moves to block 930.

At block 940, the proxy server 120 queries the threat database 124 todetermine whether the cookie (if one is included in the request 154) isincluded on the global allow cookie list or the local allow cookie listfor the requested domain. If the cookie included in the request is oneor both of those lists, then flow moves to block 935, otherwise flowmoves to block 945.

At block 930, the proxy server 120 queries the threat database 124 todetermine whether the cookie included in the request (if one isincluded) is included on the local restricted cookie list for therequested domain. If the cookie is not on the list, then flow moves toblock 935; otherwise flow moves to block 945.

At block 935, the proxy server 120 determines whether the request 154itself includes harmful material (e.g., virus, worm, or othervulnerability, malformed header, malformed cookie, etc.). If the requestincludes harmful material, then flow moves to block 945. If the requestdoes not include harmful material, then flow moves to block 822 of FIG.8.

FIG. 11 is a flow diagram illustrating exemplary operations fordetermining whether the request includes harmful material according toone embodiment. At block 1115, the proxy server 120 determines whetherthe cookie of the request (if one is included) or other header ismalformed. Malformed cookies or headers serve as an indication that therequest 154 is not from a human user and is a likely indication ofsuspicious activity. An example of a malformed header occurs if an HTTPGET request does not contain a URL and an HTTP version string. If therequest includes a malformed cookie or header, then flow moves to block945 of FIG. 9; otherwise flow moves to block 1115.

At block 1115, the proxy server 120 determines whether the URL of therequest is malformed or contains a known threat signature. To determineif the request includes a known threat signature, the proxy server 120accesses known threat signatures in the threat database 124 to determinewhether the URL matches a threat signature. If the URL is malformed ormatches a known threat signature, then flow moves to block 945 of FIG.9; otherwise flow moves to block 1125.

At block 1125, the proxy server 120 determines whether the request 154is a POST request. If the request 154 is not a POST request, then flowmoves to block 822 of FIG. 8. If the request is a POST, the proxy server120 analyzes the material attempted to be POSTED to determine whether itcontains a threat. Thus, if the request 154 is a POST request, then flowmoves to block 1135 where the proxy server 120 caches the contentattempted to be POSTed. Flow then moves to block 1140, where the proxyserver 120 determines whether the contents attempted to be POSTedcontains a threat. For example, the proxy server 120 accesses the threatsignatures in the threat database 124 to determine whether the contentsattempted to be POSTed match a known threat signature. If the contentsattempted to be POSTed match a known threat signature, then flow movesto block 945 of FIG. 9; otherwise flow moves to block 1145 where theproxy server 120 inserts the POST contents back into the request streamand flow moves to block 822 of FIG. 8.

Referring back to FIG. 9, at block 945 the proxy server 120 forms arequest having a block page that alerts the user of the client device110A that access to the requested content has been blocked. The blockpage is made of various elements that are customized based on thecharacteristics of the website and/or characteristics of the visitor. Insome embodiments, the block page includes the logo of the web site thevisitor was attempting to visit, a thumbnail screenshot of the websitethe visitor was attempting to visit, or a full-sized screenshot of thewebsite the visitor was attempting to visit. In some embodiments, theblock page appears as a floating HTML element directly over the websitethe visitor was trying to visit. In some embodiments, the block pageappears as a frame or HTML element immediately adjacent to the websitethe visitor was trying to visit. In embodiments where a website's logoor screenshots of the website are used, the logo or screenshots may becached and stored in the cache database 122 or they may be calculated inreal time as the visitor requests access to the website.

In some embodiments, the information on the block page is customizedwith information about the requested website. For example, the text onthe block page may include the name of the requested website from whichthe visitor is being blocked (e.g., “The owner of www.example.com haschosen to block potentially dangerous visitors.”). The name of thewebsite may be entered by the owner of the site in advance and stored,or it may be calculated from the URL or other header information in therequest 154. Other information about the requested website may also bedetermined including custom instructions, color choices, font choices,layout or design of the block page, and positioning of the block pagerelative to the website (e.g., covering the whole page, as a bar at thetop of the page, as a bar on the side of the page, etc.). In someembodiments the block page settings are specified by the domain owner ofthe website or set by an administrator of the service.

In some embodiments, in addition to or in place of the characteristicsof the website, the block page is customized based on a set of one ormore characteristics of the particular visitor being blocked. Thesecharacteristics may include one or more of the following: the IP addressof the request 154, the referring URL, the user-agent of the request,the visitor's operating system, the visitor's connection speed, thepreferred language of the visitor, any cookies on the visitor'sbrowsers, the reason the visitor was blocked, and other characteristicsof the visitor (e.g., the fonts installed on the browser, the languagessupported by the browser, whether the browser executes JavaScript,etc.). From these initial characteristics, additional characteristicscan be determined. For example, the IP address of the request can bereferenced to a database to determine the country of origin the requestis from. The country of origin of the visitor can then be referencedagainst a geolocation database in order to lookup languagespredominately spoken in that country. This, combined with the preferredlanguage characteristic, may be used by the proxy server 120 in order todeliver text of the block page written in one or more languages likelyto be spoken by the visitor. Different translations of the text of theblock page may be stored by the proxy server 120 (or other centralserver) or the translation may be done through language processingsoftware by the proxy server 120 in real time.

Beyond adjusting the language of the text of the block page, thelocation of the visitor may be used to deliver links to products to helpeliminate the underlying problem that caused the visitor to be blocked.For example, if the visitor is from France and is being blocked becauseof an underlying virus infection, the block page may include links toanti-virus products that are targeted to those solutions that haveproducts designed for a French audience. As another example, the blockpage may also include a telephone number specific to the country of thevisitor that they can call to get help solving the problem that causedthem to be blocked. Similarly, the block page may include links orsuggestions for anti-virus products specific to the operating system ofthe visitor.

In addition to the location of the visitor, other information may beused to customize both the links to solutions that are provided as wellas the text displayed on the block page. For example, if the reason thatthe visitor was blocked was because of a virus infection, the system maychange the text to provide information about the virus infection (e.g.,“You were blocked because it appears you have a virus running on yourmachine that is sending out spam email,” or “You were blocked because itappears you have a the MyDoom virus running on your computer.”). Thelinks to solution providers may also be targeted based on the underlyingproblem that caused the visitor to be blocked. For example, if oneanti-virus company is particular adept at removing a particular type ofvirus, links to that company's products may be more likely to bedisplayed, or may be displayed more prominently (e.g., in bold, with ahighlight, or at the top of a list of multiple other solutionproviders).

The text and links to solution providers may also be customized based oncharacteristics of the IP address of the request. For example, the proxyserver 120 can look up the reverse DNS and whois entry for the IPaddress. This information can be used in order to attempt to determinethe owner of the IP address. This owner information can be used in orderto determine additional characteristics of the visitor. For example, ifa reverse DNS entry displays that the reverse DNS for an IP address isnode1.example.net, the proxy server 120 can compare this entry againstother known reverse DNS entries and can determine the type of connection(e.g., Internet provider, corporate, commercial, etc.). In this case,the system could drop the subdomains of the reverse DNS entry one by oneand check if the remainder of the entry is similar to anything stored inthe system. In this example, the proxy server 120 may learn thatexample.net is an Internet service provider. The proxy server 120 cancharacterize connections into various categories based on the reverseDNS and whois information. These categories may include, but are notlimited to, residential Internet connections, business Internetconnections, government Internet connections, school Internetconnections, etc.

In some embodiments, the text and/or solution suggestions are customizedbased on both the organization identified as the owner of the visitor'sIP address in the reverse DNS and whois records as well asmeta-information about the organization. For example, a visitor from aresidential Internet provider may receive links to personal computeranti-virus. As another example, a visitor from a corporate network mayreceive information (e.g., a link to a white paper or other document)that includes a description of the underlying problem and instructionsto forward it to the corporate network provider. The information may becustomized with details of the reason for the visitor having beenblocked as well as information or advertisements for solution providersthat can protect users of the network in the future, as well asstatistics regarding the underlying problem (e.g., the number of usersthat have been known to experience the same underlying problem, thenumber of threats detected in their network, comparison with othersimilar networks, etc.).

Vulnerability solution providers may request to target only certainkinds of visitors. For example, an anti-virus company may request thatlinks for its products may only show up for visitors with a certain setof characteristics (e.g., display an ad for solution 1 from provider Aif the visitor is from the Korea, Japan, or China, is running theWindows operating system, is blocked between 12:00 and 1:00 on Mar. 1,2010, and is blocked because of a MyDoom virus infection). In someembodiments the block page includes all the solutions that match thecharacteristics requested, while in other embodiments the block pageincludes only a limited set of providers that match the requestedcharacteristics. In the latter case, the different solution providersmay be ranked by how closely the visitor matches the characteristicsthey described and/or by how much the solution provider has offered topay for the solution provider's link to be displayed. Links to solutionproviders may be text links, logos of the solution providers, some otherkind of picture, a telephone number, or a combination of the above. Theblock page may also include a description of the products or serviceoffered by the solution provider.

In some embodiments, in addition or in place of the description and/orlinks to solution providers, the block page may also include a mechanismfor the visitor to dismiss the block page and continue on to therequested content. This is sometimes referred herein as a “dismissmechanism.” The dismiss mechanism may be a link to close the block page;an image, audio, or other form of CAPTCHA; a ping back from anti-virussoftware after it has scanned or cleaned the visitor's machine, anoverride code or password provided to customers, or some other dismissmechanism. In the case of the link, if it is clicked then the block pagewould be dismissed and the visitor would be allowed on to the actualsite. It should be understood in some embodiments, the block page doesnot include a dismiss mechanism for the visitor to dismiss it.

In the case of the dismissal of the block page requiring a CAPTCHA, theproxy server 120 records the input of the user in response to agraphical image, audio recording, math problem, or something else. Ifthe visitor responds correctly without too many mistakes, the proxyserver 120 will dismiss the block page and allow the visitor to retrievethe requested content. In some embodiments, customers are provided anoverride code or password that can be entered by the customer uponencountering a block page. If successfully entered, the blocking of evenmalicious requests would be allowed.

In the case of the dismissal of the block page requiring a ping backfrom anti-virus software after it has scanned or cleaned the visitor'sclient device, the visitor may be prompted by the block page to scantheir client device with anti-virus software or download an anti-virussoftware program. If the anti-virus software is run and no infectionsare found, the anti-virus software may send a request to the proxyserver 120 (or other computing device of the service) with a uniquesystem clean code, the IP address of the visitor, the cookie of thevisitor, and other characteristics to identify the visitor's clientdevice (e.g., the fonts installed, the operating system running,installed plugins, installed browsers, MAC address(es) of networkcard(s), email address of the visitor, etc.). This indicates to theproxy server 120 that the client device is free from viruses or othersoftware vulnerabilities and is a candidate to be removed from theglobal restricted IP address list if that IP address is on that list. Asanother example, after completing a system scan, the anti-virus softwaremay provide the visitor with a unique code that the visitor can enter onthe block page in order to dismiss it. Alternatively, if the anti-virussoftware finds a problem or virus infection, the anti-virus software mayoffer the visitor options to remove the problem. The anti-virus softwaredoes not ping back to the proxy server 120 or provide a code to dismissthe block page until after the visitor's client device has beenconfirmed as clean.

The system clean code is a unique identifier (e.g., a string ofcharacters) that is supplied by a trusted third party (e.g., anti-virussoftware company) certifying that the underlying problem has beenremedied. The system clean code could be from a list of codes stored bythe trusted third party or application (e.g., an anti-virus application)and issued one at a time until they are exhausted. At that time, theanti-virus company could request more codes from the application.Alternatively, the code could be from a list stored by the applicationand retrieved through an API interface by the trusted third party orapplication (e.g., anti-virus application). Codes could be requested viathe API after the visitor's client device is certified clean.Alternatively, the code is a hash of unique attributes of the clientdevice that has been cleaned by the trusted third party or application(e.g., anti-virus application). In this case, the code could be astandard hash (e.g., MD5, SHA1, SHA256, etc.) of some combination ofattributes such as the disinfected client device's browser cookie, IPaddress, or other attributes transmitted during a connection. Theseattributes may be combined with a secret salt value in order to makethem secure. In this case, the anti-virus application may generate thecode itself, or may transmit the attributes to the system for the codeto be generated centrally. In any case, once the code is entered therecord of the visitor infected can be cleared and the visitor would nolonger be blocked. The block may be reinstituted if there is evidencethe visitor has become reinfected.

In some embodiments the domain owners select whether a dismiss mechanismis included in the block page and how that dismiss mechanism operates,while in other embodiments the service selects whether a dismissmechanism is included in the block page and how that dismiss mechanismoperates. In some embodiments, the proxy server 120 uses the visitor'scharacteristics to determine whether to include a dismiss mechanism inthe block page. For example, a visitor may be blocked because of a virusinfection on the visitor's machine. The proxy server 120 may look up thethreat score the particular visitor represents. Threat scores may berepresented on an ordered scale. The proxy server 120 can then look tosee the preferences of the particular website. For example, the domainowner may specify that visitors with a threat score of 5 or less get asimple link to dismiss the block page; visitors with threat scores from6-10 must pass a CAPTCHA before they may dismiss the block page;visitors with a threat score from 11-20 may not dismiss the block pageunless they use anti-virus software to scan their machine and remove theunderlying infection; and visitors with a threat score over 20 may notdismiss the block page at all.

In some embodiments, in addition to or in place of a CAPTCHA, visitorsare required to provide an email address, phone number, or othercommunication address, and must respond to an email, phone call, textmessage, etc. For example, the visitor may be prompted to click on alink in an email, enter a code through the telephone, send a codethrough a text message, etc. Only after the identifying prompt has beencompleted will the block page be dismissed.

In some embodiments, if a visitor dismisses a block page, the proxyserver 120 records the event in the visitor statistics. In some cases,the proxy server 120 removes the visitor from the global restricted IPaddress list subsequent requests for the same content will not triggeranother block page. In some embodiments, a cookie with a uniqueidentifier is stored on the visitor's browser after dismissing a blockpage. When the visitor makes subsequent requests, the proxy server 120reads the cookie and allows the visitor access without triggering theblock page. The cookie may include an expiration time in its value or itmay be set to automatically expire. The expiration time may bedetermined by the system default, based on the characteristics of theparticular threat (e.g., what is the threat score of the visitor). Whenthe cookie expires, the visitor will once again be blocked unless therestricted listing has been removed.

FIGS. 10A-B are exemplary block pages according to one embodiment. Asillustrated in FIG. 10A, the block page 1000 includes the block pageribbon 1005 which is overlaid on the background image or content 1010 ofthe requested content's website. For example, the background image orcontent 1010 may be a logo of the requested website, a thumbnailscreenshot of the requested website, or a full-sized screenshot of therequested website. The background image or content 1010 may besemi-transparent as compared with the block page ribbon 1005. The blockpage ribbon 1005 includes the explanatory text 1015 which explains whythe visitor has been blocked from the website. The dismiss mechanism1020 is a form of a CAPTCHA where a user needs to enter in a word thathas been scrambled such that it is difficult for a non-human visitor toread and subsequently enter. The anti-virus solutions or disinfectinstructions 1025 may include instructions to fix the underlying problemthat caused the visitor to be blocked and/or links for the visitor todownload anti-virus software to clean their client device. FIG. 10B isanother example of a block page that is similar to FIG. 10A with theexception of the dismiss mechanism. As illustrated in FIG. 10B, thedismiss mechanism 1040 includes a field for the visitor to enter in acode that is provided by an anti-virus software program after completinga scan and fixing any errors found by the anti-virus software program.

Referring back to FIG. 9, flow moves from block 945 to block 950 wherethe proxy server 120 determines whether the visitor has successfullyoverridden the block page. For example, with reference to FIG. 10A,whether the correct CAPTCHA input was entered in the dismiss mechanism1020; and with reference to FIG. 10B, whether a valid override blockpassword was entered in the dismiss mechanism 1040. If the visitor hasnot successfully overridden the block page, then flow moves to block 955the proxy server 120 flushes its cache and the visitor will remainblocked. However, if the visitor successfully overrides the block page,then flow moves to block 960 where the proxy server 120 rebuilds theoriginal request and inserts it into the request stream. For example,with reference to claim 1, the proxy server 120 rebuilds the request154. Flow moves from block 960 to block 822 of FIG. 8.

With reference back to FIG. 8, after determining that the request and/orthe visitor is not a threat at block 815, flow moves to block 822 wherethe proxy server 120 determines whether the request 154 includes anoffline browsing cookie that is set. An offline browsing cookie is acookie set by the proxy server 120, or other proxy servers of theservice, when the origin server returns a server offline error (e.g., anHTTP status code 500 error). Customers can also set offline browsingcookies if they want to place their origin servers as operating inoffline browsing mode. An offline browsing cookie may be a domain-widecookie (e.g., example.com) or specific to a particular subdomain (e.g.,news.example.com). If an offline browsing cookie is included in therequest, then flow moves to block 1620 of FIG. 16, which will bedescribed in greater detail later herein. If an offline browsing cookieis not included in the request, then flow moves to block 825.

At block 825, the proxy server 120 determines whether the request 154 isfrom a search engine or a web crawler (e.g., a crawler used to indexpages to provide faster search results). For example, the proxy server120 searches a file or database with IP addresses of known searchengines and known crawlers for the IP address of the request 154 (thefile or database may reside locally on the proxy server 120 or may belocated at a remote server). In some embodiment, the proxy server 120also checks whether the user-agent of the request matches the user-agentof the known search engine/crawler. If the request is from a searchengine or crawler and flow moves to block 835. If the request is notknown to be a search engine or crawler, then flow moves to block 830.

At block 830, the proxy server 120 determines whether the request is fora static cacheable resource. A static cacheable resource is a resourcethat is eligible for being cached in the general purpose cache of thecache 122. Typically static cacheable resources are static files (e.g.,images, videos, etc.) and do not include dynamic content (e.g., HTMLfiles). If the request is for a static cacheable resource, then flowmoves to block 840; otherwise flow moves to block 832. At block 832, theproxy server 120 queries the origin server for the requested content.For example, with reference to FIG. 1, the proxy server 120 transmitsthe request 156. Flow moves from block 832 to block 838 where the proxyserver 120 receives the response from the origin server and returns theresults to the requesting client device. For example, with reference toFIG. 1, the proxy server 120 receives the response 158 from the originserver 130 and transmits the response 162 to the client device 110. Itshould be understood that since the request was not for a human staticcacheable resource, the proxy server 120 does not cache the receivedcontent in the cache 122. In some embodiments, instead of directlyquerying the origin server as described in reference to block 832, theproxy server 120 determines whether to redirect the request directly tothe origin server instead of passing the request and response throughthe proxy server 120. In such embodiments, flow moves from block 830 toblock 1210 of FIG. 12.

Returning back to block 840, the proxy server 120 queries the cache 122(e.g., the general purpose cache) for the requested resource and flowmoves to block 845. Returning back to block 835 (the request is from asearch engine or crawler), the proxy server 120 queries the cache 122(e.g., the special purpose cache and/or the general purpose cache) forthe requested resource and flow moves to block 845. If the requestedresource is not available in the cache 122, then flow moves to bock 855.If the requested resource is available in the cache 122, then flow movesto block 850 where the proxy server 120 determines whether the cachedcontent has expired. If it has not expired, then flow moves to block 865where the proxy server 120 returns the cached copy without querying theorigin server. For example, with reference to FIG. 1, the proxy server120 accesses the cache 122 in the request and receive content operation160 and transmits the response 162 to the client device.

At block 855 (the requested resource is not in the cache 122), the proxyserver 120 queries the origin server for the requested content. Forexample, with reference to FIG. 1, the proxy server 120 transmits therequest 156 to the appropriate origin server. Flow then moves to block860. The proxy server 120 receives the response 158 from the originserver and stores a copy of the content in the cache 122 (e.g., in thegeneral purpose cache and/or the special purpose cache). In someembodiments, the proxy server 120 also assigns an expiration value forthe cached content, which may depend on the type of content cached(e.g., static content may have a longer expiration period than dynamiccontent). After caching a copy of the content, the proxy server 122 maylocally respond to future requests from search engines or known crawlersfor that content. Flow moves from block 860 to block 870 where the proxyserver 120 returns the requested content to the client device. Forexample, with reference to FIG. 1, the proxy server 120 transmits theresponse 162 to the client device.

In some embodiments, instead of directly querying the origin server asdescribed in reference to block 855, the proxy server 120 determineswhether to redirect the request directly to the origin server instead ofpassing the request and response through the proxy server 120. In suchembodiments, flow moves from block 845 to block 1210 of FIG. 12 when therequested resource is not in the cache 122.

FIG. 12 is a flow diagram illustrating exemplary operations forredirecting requests directly to origin servers according to oneembodiment. At block 1210, the proxy server 120 determines whether therequest 154 is requesting content that is known to be unsupported by theservice (those that will not pass through the proxy server 120). Forexample, video files may be unsupported by the service. In someembodiments, the service supports different content types for differentorigin servers (e.g., domain owners may subscribe to different levels ofservice where an increasing level of service corresponds with anincreasing number of content types supported by the service). If therequest is for content that is known to be unsupported by the service,then flow moves to block 1215; otherwise flow moves to block 1235.

At block 1235, the proxy server 120 determines whether the request isfor content that is known to be for a large file (i.e., greater than athreshold). In some embodiments, the proxy server 120 analyzes theheader of the file which gives the content length of the file, which canbe compared against the threshold. In some embodiments, the proxy server120 maintains a large content list for each of the origin servers 130A-Lthat indicate files that are above a certain threshold. In someembodiments, different thresholds apply to different origin servers(e.g., domain owners may subscribe to different levels of service wherean increasing level of service corresponds with an increasingthreshold). If the request is not for a known large file type, then flowmoves to block 1215; otherwise flow moves to block 1240 where therequest is handled by the proxy server 120 as previously described.

At 1215, the proxy server 120 transmits a redirection to the clientdevice 110 that redirects the client device to a subdomain of the domainof the request. For example, the proxy server 120 transmits the response162 to the requesting client device 110 that includes an HTTP responsestatus code 301, 302, 303, 305, 307, or other redirection status codethat indicates that the client device should be redirected to adifferent subdomain, or a page with a meta redirect and/or a scriptinstructing the client network application to reload the page in anothersubdomain. The zone file entry for the subdomain will point directly tothe origin server and not the proxy server 120. By way of example,assuming the request was for content of the domain example.com, theproxy server 120 transmits a redirection to a subdomain of example.com(e.g., redirect.example.com). Flow moves from block 1215 to block 1220.

At block 1220, the browser of the client device queries the DNS system140 to determine the IP address of the subdomain. Next, the clientdevice receives the IP address of the subdomain from the DNS system. TheIP address will point directly to the origin server and not the proxyserver 120. Flow then moves to block 1230, where the client devicetransmits the redirected request 170 to the origin server and the originserver 130 responds accordingly (e.g., issuing the direct response 172to the requesting client device).

Validating Visitor Threat

Since IP addresses may not be assigned statically (e.g., they may beassigned dynamically through DHCP), may change who they are assigned toover time, and are subject to being hijacked or spoofed, it is possiblefor an IP address of a request to be listed on a restricted list (e.g.,the global restricted IP address list) even though the client devicethat is currently associated with that IP address was not responsiblefor that IP address being added to the restricted list. For example, aclient device that receives a dynamic IP address in a public place(e.g., a coffee shop, a library, or other Wi-Fi spot) may receive an IPaddress that has been listed on a restricted list due to actionsperformed by a previous client device assigned that IP address. In otherwords, since a particular IP address is not necessarily tied to aparticular client device, there is a possibility that false positives ofthreats may occur when checking the IP address restricted lists.

In some embodiments, to determine whether the client device isresponsible for the IP address being listed on a restricted list, ahierarchical threat checking mechanism is used such that the clientdevice is redirected to a validating domain server for a validatingdomain designed to create and read cookies for the validating domainresponsive to detecting that the IP address of the request is includedon a restricted list (e.g., the global restricted IP address list or thelocal restricted IP address list). Each cookie includes an indicationwhether the corresponding client network application has previously beendetermined to be participating in suspicious activities or has beenproven to be a human user and not a bot.

With reference to FIG. 1, after determining that the request 154 is athreat due to its IP address being listed on a restricted list (e.g.,the global restricted IP address list of the threat database 124), theproxy server 120 redirects the requesting client device to thevalidating domain, which is served by the validating domain server 180.The client device makes the request 186 to the validating domain server180 with a cookie for the validating domain (if one exists on the clientdevice). The validating domain server 180 determines, through reading acookie of the request 186 (if one is included), whether the cookieoverrides the listing of the IP address on the restricted list. While insome embodiments, if the cookie overrides the listing of the IP addressbeing included on the restricted list, the response 188 will redirectthe visitor to the origin server; in other embodiments the response 188will redirect the client device to issue the request 154 to the proxyserver 120 which will not treat the request as being a threat (at leastdue to its IP address being included on a restricted list). If thecookie does not override the listing of the IP address, in someembodiments the response 188 includes a block page indicating that thevisitor has been blocked and may include a dismiss mechanism.

FIG. 13 is a flow diagram illustrating exemplary operations forvalidating whether a request should be subject to restriction afterdetermining that its IP address is listed on a restricted list accordingto one embodiment. In some embodiments, the operations described inreference to FIG. 13 are not performed when a request includes a validcustomer bypass cookie as previously described with reference to block910 of FIG. 9.

At block 1310, the proxy server 120 has determined that the IP addressof the request 154 is on a restricted list (e.g., the global restrictedIP address list). Flow then moves to block 1315 and the proxy server 120redirects the requesting client device to a page within the validatingdomain. For example, the response 162 includes an HTTP response statuscode 301, 302, 303, 305, 307, or other redirection status code thatindicates a redirection to the validating domain, or a page with a metaredirect and/or a script instructing the browser to load a page in thevalidating domain. Flow moves from block 1315 to 1320.

At block 1320, the validating domain server 180 receives the request 186from the requesting client device. Flow then moves to block 1325 wherethe validating domain server 180 determines whether the request 186includes a cookie for the validating domain. If it does not include acookie, then flow moves to block 1330 where the validating domain server180 creates a cookie for the requesting client device. After recordingthe request and the validating domain cookie to the global cookiedatabase 185 in block 1335, flow then moves to block 1340 where thevalidating domain server 180 detects characteristics about the clientnetwork application of the requesting client device. For example, thevalidating domain server 180 creates a client network applicationfingerprint based on one or more of the following: whether the clientnetwork application loads images; whether the client network applicationexecutes JavaScript; the type of network application (e.g., browser nameand version); the operating system running the client networkapplication; the fonts installed on the client network application; thelanguages supported by the client network application; whether theclient network application supports plugins (e.g., flash plugins);whether the client network application stores cookies; and whether theclient network application responds from the same IP address for variousprotocol requests.

Flow moves from block 1340 to block 1345, where the validating domainserver 180 returns a block page to the requesting client device. Forexample, the response 188 includes a block page indicating that thevisitor has been blocked and may include a dismiss mechanism asdescribed in reference to FIGS. 9, 10, and abb.

If the request 186 includes a cookie, then flow moves from block 1325 toblock 1350 where the validating domain server 180 reads the cookie. Thevalidating domain server 180 then records the request and the cookie inthe global cookie database 185 and flow moves to block 1355.

The validating domain server 180 determines whether the cookie in therequest overrides the listing of the IP address on the restricted listat block 1355. For example, a cookie may include information, orindicate with a code or unique token which is associated with a recordof the visitor, that the visitor's client device is clean of viruses (asverified by an anti-virus application), that one or more customers havewhite-listed the visitor and/or marked the visitor as not a threat, thatthe behavior of the visitor is consistent with a human user and not abot, etc. For example, the cookie may indicate that the user of therequesting client device has previously proved that he or she is a humanuser and is not a bot (e.g., by dismissing a block page using a dismissmechanism). If the cookie overrides the listing of the IP address on therestricted list, then flow moves to block 1360, otherwise flow moves toblock 1340. At block 1360, the validating domain server 180 redirectsthe requesting client device to the origin server such that the originserver responds to the request. In other embodiments, instead ofdirecting the requesting client device to the origin server, thevalidating domain server 180 causes the IP address to be removed fromthe restricted list and redirects the requesting client device back tothe proxy server 120.

Tarpitting Visitors that are Threats

In some embodiments, responsive to the proxy server 120 determining thata visitor requesting content has been identified as a threat (e.g., theIP address of the request is included on the global restricted IPaddress list and/or the local restricted IP address list) and optionallyhas been verified as a threat, the proxy server 120 tarpits the visitorincluding reducing the speed at which it processes the requests andresponses for the session such that the connection remains open and/orcreates a set of one or more false links in the response in order tooccupy bots causing them to waste time and resources in following linksthat do not exist.

FIG. 14 is a flow diagram illustrating exemplary operations fortarpitting a visitor according to one embodiment. At block 1410, theproxy server 120 receives a request and determines that the visitor is athreat (e.g., the IP address of the request is included on a restrictedIP address list). Flow then moves to block 1415, where the proxy server120 reduces the speed at which the request and any response will beprocessed, while keeping the connection open (thus preventing theconnection from timing out). For example, the proxy server 120 turnsdown the number of bytes per second that it delivers for thisconnection. Flow moves from block 1415 to block 1420. Since each clientnetwork application has a limited number of connections it can make tothe Internet, slowing down a connection that is known to be performingsuspicious activities both prevents that client network application fromparticipating in suspicious activities to that website as well aslimiting the total number of suspicious activities that cansimultaneously be performed by that client network application. In otherwords, the amount of time waiting for the throttled connection reducesthe amount of time that a different connection to a different websitecould be opened by the client network application to perform suspiciousactivity on that different website.

At block 1420, the proxy server 120 generates a response with a numberof false links to domains that are protected by the service. Flow thenmoves to block 1425, where the proxy server 120 transmits the page tothe visitor with the false links (which is processed at the reducedspeed). The false links are included in the response such that botvisitors will be occupied by following the false links, which will besubject to the reduced processing restriction as the original request.The false links correspond with false pages that can either be randomlygenerated, generated based on cached content (e.g., the false links maypoint to content stored in the cache), and/or reproduced from otherpages, which may or may not be pages protected by the service. It shouldbe understood that the links of the page are replaced with false linksthat point to false content and so on such that the process ofgenerating false links continues indefinitely for each new request andare self referential (thus there is no path out of the labyrinth). Thus,a labyrinth of links that are protected by the service (e.g., requestswill be directed to the proxy server) is created into which maliciousbots (e.g., email harvesters, etc.) can be effectively trapped. Itshould be understood that the amount of time the bot spends in followingthe false links and the amount of time waiting for each request andresponse to be processed reduces the ability of the bot to performsuspicious activity on different Internet sites, which may or may not beprotected by the service. Bots which fall into this tarpit have adifficult time escaping because new false links and false pages areindefinitely generated.

In some embodiments, instead of blocking access to requested content bydelivering a false page, the connection speed is reduced yet therequested content is still delivered. This has the benefit ofeliminating false positives since, although the connection speed isreduced for a given connection, a legitimate user may still receive therequested content. In some embodiments, the proxy server 120 adds one ormore false links that are hidden from human users but capable of beingread and followed by bots.

Content can be hidden from human users in a number of ways. For example,CSS can be used to mark a particular portion of code as havingdisplay=none or display=hidden. Alternatively, CSS can be used to movethe link off the page (<a href=“http://www.example.com/”style=“position:absolute; left: −250px; top: −250px;”>Some Hidden Link</a>). A tag canbe included with no contents (e.g., <ahref=“http://www.example.com”></a>). Content can be included in acomment (e.g., <!--<a href=“www.example.com”>Some HiddenLink</a><//--!>). A script can be used to hide a link after it has beenrendered. A link can be wrapped around a single-pixel, or extremelysmall, image. It should be understood that these are exemplarytechniques and other techniques may be used to include links in a pagethat is visible to bots but hidden from humans.

In addition to the requested content, the proxy server 120 may also adda mechanism for the user at the requesting client device to prove theirlegitimacy and thereby increase their connection speed (i.e., to breakout of the tarpit). For example, similar to the dismiss mechanism of theblock page, the proxy server 120 may add a mechanism to the response (oras a separate response) such as a CAPTCHA challenge for the user of therequested client device to answer, which if performed successfully, mayremove the throttling of the connection speed.

In other embodiments, in addition to slowing down the connection forsuspicious users, the proxy server 120 responds with false information.For example, instead of responding with the requested content at simplya reduced speed, the proxy server 120 responds with false information ofno value to the requesting client device. Thus in addition to slowingdown the connection for suspicious users, the content that they receiveis of no value. In such embodiments, the proxy server 120 respondslocally and does not transmit a request to the origin server. Inaddition to or in place of the false information, the proxy server 120may respond with a mechanism for the user at the requesting clientdevice to prove their legitimacy and thereby increase their connectionspeed. For example, similar to the dismiss mechanism of the block page,the proxy server 120 may add a mechanism to the response (or as aseparate response) such as a CAPTCHA challenge for the user of therequested client device to answer, which if performed successfully, mayremove the throttling of the connection speed.

The bandwidth allocated to visitors can also be throttled based onserver load such that in periods of relatively high load on the proxyserver, the visitors with relatively higher threat scores and/orrequesting false content (e.g., they are tarpitted) have their bandwidththrottled while the bandwidth allocated to visitors with relativelylower threat scores and/or not requesting false content is maintained.

In some embodiments, in addition to trapping malicious bots, the proxyserver 120 can record recent queries to the labyrinth and report thenumber of malicious crawlers it currently occupies as well as thebandwidth and time it has caused them to waste. In addition, the proxyserver 120 can leverage that a visitor is caught in the tarpit asevidence to support the conclusion that it is malicious.

Response Related Actions

With reference to FIG. 1, after receiving the response 158 from anorigin server 130, the proxy server analyzes the response (at theanalyzing response operation 166) and performs a set of one or moreresponse related actions to perform based on the results of theanalyzing.

FIG. 15 is a flow diagram illustrating exemplary operations forperforming response related actions according to one embodiment. Atblock 1510, the proxy server 120 receives the response 158 from anorigin server 130. Flow then moves to block 1515 where the proxy server120 analyzes the response 158 (as indicated by the analyze responseoperation 166 of FIG. 1). Control moves from block 1515 to block 1520.

At block 1520, the proxy server 120 determines whether the response 158includes an error (e.g., an HTTP status code 4XX client error or 5XXserver error). If the response 158 includes an error, then flow moves toblock 1525. If the response 158 does not include an error, then flowmoves to block 1710 of FIG. 17. At block 1525, the proxy server 120determines whether the error is an indication that the origin server isoffline (e.g., the error is an HTTP status code 500 error). If yes, thenflow moves to block 1615, which will be described in greater detail withrespect to FIG. 16. If no, then flow moves to block 1530.

At block 1530, the proxy server 120 determines whether the error is anindication that the resource is unavailable (e.g., the error is an HTTPstatus code 404 error) If it is not, then flow moves to block 1535 wherethe proxy server 120 transmits the response 162 to the requesting clientdevice with the error supplied by the origin server. If the error is aresource unavailable error, then flow moves to block 1540 where theproxy server 120 transmits the response 162 to the requesting clientdevice with a custom error page. For example, in some embodiments, theproxy server 120 creates a custom error page based on one or more of thefollowing: the location (e.g., country) of the visitor who triggered theerror; the location of the origin server; the language of the visitorwho triggered the error; any cached content on the website where theerror occurred, including the page that may have been cached before theerror occurred or other pages on the site that give an overall contextto the site generally (e.g., the type of site (e.g., sports, news,weather, entertainment, etc.)); a list of links or terms provided by thecustomer; the list of the most accessed pages elsewhere on the websitedetermined by other visitors; any terms that can be parsed from therequest URL or POST; and any terms that can be parsed from the referrerURL. Based on one or more of these factors, the error page may includelinks to other pages on the website or on other websites that would beof interest to the visitor. In some embodiments, the links are sponsoredto advertisers looking to target individuals requesting particularcontent. In some embodiments, if the originally requested content hasmoved locations (e.g., it is now available at a different URL), thecustom error page includes a link to a new location of the originallyrequested content or automatically redirects the visitor to the newlocation of the originally requested content. In some embodiments, theproxy server determines that requested content has moved locations bycomparing a hash of the original content with hashes of other contentfrom the same origin server when an error message occurs.

Origin Server Offline Handling

In some embodiments, the proxy server 120 serves cached content (whenavailable) to requesting client devices when the origin server isoffline. In addition, if the requested content is HTML content or othercontent that can be modified, the proxy server 120 adds an automaticserver query script to the cached content to cause the client networkapplication of the visitor to automatically and periodically determinewhether the origin server is online (e.g., by pinging the originserver). In some embodiments, to reduce the load on the origin server,the automatic server query script queries the proxy server 120periodically and the proxy server 120 periodically queries the originserver. In addition, the proxy server 120 does not immediately ping theorigin server upon receipt of each query request it receives fromexecuting automatic server query scripts. Rather, the proxy server 120maintains an independent origin server ping timer to determine when toping the origin server such that the number of times the origin serveris pinged is reduced. In some embodiments, the origin server ping timeris specific to the entire domain represented by the origin server (e.g.,example.com) and is not limited to a specific resource of the domain(e.g., example.com/example.html), while in other embodiments there is aseparate origin server ping timer for each resource.

In other embodiments, the automatic server query script, when executed,directly queries the origin server to determine whether it is online.For example, the automatic server query script may check to determinewhether the origin server is online by pinging it.

FIG. 16 is a flow diagram illustrating exemplary operations performed bythe proxy server 120 when responding to server offline errors accordingto one embodiment. At block 1610, the proxy server 120 detects that theorigin server is offline. For example, the proxy server 120 receives theresponse 158 which includes an error code that indicates that the originserver is offline. Flow then moves to block 1615, where the proxy server120 determines whether the requested resource is available in cache. Forexample, the proxy server 120 queries the cache 122 for the resource. Ifit is not available, then flow moves to block 1630 where the proxyserver 1630 returns an error to the requesting client device thatindicates that the origin server is offline and includes an offlinebrowsing cookie for the requested domain. If the resource is availablein cache, then flow moves to block 1620.

At block 1620, the proxy server 120 determines whether the requestedresource is an HTML file (e.g., the proxy server 120 examines the headerof the request to determine whether it is an HTML file). If therequested resource is an HTML file, then flow moves to block 1635,otherwise flow moves to block 1680 where the proxy server 120 returnsthe cached resource to the requesting client device in the response 162.The cached resource is associated with a TTL (time-to-live) value set bythe proxy server 120.

At block 1635, the proxy server 120 rewrites the HTML page to indicateto the user that it is viewing a cached copy of the page. This mayinclude the time and date at which the cached version was created. Flowthen moves to block 1640, where the proxy server 120 adds an automaticserver query script to the HTML page that automatically pings the proxyserver 120 at periodic intervals to query the origin server to determinewhether it is online. The script will execute as long as the page isopen on the client device.

Flow then moves to block 1645, where the proxy server 120 adds anoffline browsing cookie. Next, flow moves to block 1650 and the responsewith the modified HTML page and the offline browsing cookie is sent tothe requesting client device.

Flow moves from 1650 to 1655 where the proxy server 120 receives a queryrequest from the script added to the HTML page of the cached content.Responsive to receiving the query request, the proxy server 120determines whether to query the origin server to determine whether it isonline at block 1660. For example, in some embodiments, the cachedresource is associated with a TTL (time-to-live) value set by the proxyserver 120 which serves as an origin server ping timer. When the TTLvalue has expired, flow moves to block 1665 where the proxy server 120queries the origin server to determine whether it is online (e.g., bypinging the origin server). If the TTL value has not expired (thus it isnot time for the proxy server 120 to query the origin server), flowremains at block 1660.

At block 1665, the proxy server 120 determines whether the origin serveris online (e.g., whether the proxy server 120 receives a response fromthe origin server). If the origin server is online, then flow moves toblock 1670 where the proxy server 120 deletes the offline browsingcookie. The proxy server 120 may also transmit an offline browsingcookie to the requesting client device that has a past expiration datesuch that the next time a user makes a request for a page within thedomain corresponding to the cookie, the client network application willdetermine that the offline browsing cookie has expired and will removeit. If the origin server is offline (e.g., it did not respond to theping), then flow moves to block 1675 where the proxy server 120 resetsthe origin server ping timer. Flow moves from block 1675 back to block1655.

Modifying the Content of the Response

In some embodiments, the proxy server 120 modifies the content of theresponse 162 before delivering it to a client device 110. The content tobe modified may originate from the origin servers and/or be located inthe cache 122. Different types of content may be modified different inembodiments. For example, content that poses a threat to a client devicemay be removed from the resource. As another example, email addressesincluded in the content may be scrambled such that they will bedisplayed on the screen of the client device but will not be readablefrom the source of the page, thus preventing the email address frombeing harvested by an email harvesting program. As another example,domain owners may define rules that indicate that certain objectsincluded in content are to be modified (e.g., excluded from the contentin the response, obfuscated such that it will be displayed andunderstood by a human user but will not be readable from the source ofpage, etc.) for certain visitors based on one or more characteristics ofthe visitor (“modified rules”).

In some embodiments, the proxy server does not modify responses based oncertain visitor characteristics. For example, if the request includes avalid customer bypass cookie, the proxy server 120 will not modify theresponse.

FIGS. 17A-B are flow diagrams illustrating exemplary operationsperformed by the proxy server for determining whether and how to modifythe content of a response according to one embodiment. In oneembodiment, the operations described in FIGS. 17A-B start from block1520 of FIG. 15 (e.g., the response does not indicate an error messagesuch as a resource unavailable error or server offline error).

At block 1710, the proxy server 120 determines whether the requestedresource is an HTML page (e.g., the proxy server 120 analyzes the headerof the request to determine the type of resource). If the requestedresource is not an HTML page, then flow moves to block 1765. If therequested resource is an HTML page, then flow moves to block 1715.

At block 1715, the proxy server 120 scans the HTML page for modificationtokens. A modification token is an identifier that indicates that thecontent represented by that modification token is to be modified orremoved from the HTML page. There are different types of modificationstokens. Exemplary types of modification tokens include potential threatto a visitor, obfuscation, SSDM (server side defined modification), andadvertisement tokens. By way of specific example, if email addresses areautomatically obfuscated, a modification token of a type obfuscation isan email address that meets the following pattern: [at least onecharacter] @ [at least one character].[at least two characters]. Othermodification tokens may be described with tags. For example, an SSDMmodification token may identified with an opening tag and ending tag(e.g., <!--SSDM--> and <!--/SSDM-->). Other modification tokens canidentify phone numbers, instant messenger IDs, street addresses, linksto other websites, birthdates, social security numbers, IP addresses,credit card numbers, account usernames, etc. In one embodiment,modification token definitions, which define how tokens are identifiedand their type, are stored in a database or other data structureavailable to the proxy server 120. Flow then moves to block 1720 wherethe proxy server 120 determines whether the HTML includes a modificationtoken. If it does, then flow moves to block 1725, otherwise flow movesto block 1730. At 1730, the proxy server 120 determines whether the endof the content has been reached. If it has, then flow moves to 1735where the response is transmitted to the client device. If it has not,then flow moves back to block 1715 where the proxy server 120 continuesto scan the HTML page for tokens.

At block 1725, the proxy server 120 determines whether the modificationtoken is a type that is a threat to a visitor. For example, amodification token that may be a threat to the visitor is an element onthe page that could harm the visitor such as a virus, worm, malware,adware, etc. If the modification token is of a type that is a threat toa visitor, then flow moves to block 1740 where the proxy server 120modifies the HTML page to remove the content corresponding to thattoken. Flow moves from block 1740 back to block 1730.

If the modification token is not a type that is a threat to a visitor,then flow moves to block 1745 where the proxy server 120 determineswhether the modification token is an obfuscation type (e.g., an emailaddress that is to be obfuscated). If the modification token is anobfuscation type, then flow moves to block 1750 where the contentcorresponding to the modification token is obfuscated by replacing itwith an obfuscation script, which when executed (e.g., upon the pageloading in the client network application), generates the data such thatit will be displayed on the rendered page but will not be readable by abot (e.g., the content is not included in the page source). By way ofspecific example regarding email addresses, an email address is replacedwith an obfuscation script that, when executed, hides that email addressfrom automated bots (the email address will not appear in the pagesource) but generates the email address to be displayed to the user inthe rendered page. In some embodiments, the obfuscation script alsoencodes the email address as displayed to the user with a mailtoattribute such that when clicked by the visitor, their email programwill launch (if it is not already launched) and a new email messagewindow will be created that is addressed to that email address.

FIG. 18 is a flow diagram illustrating exemplary operations forobfuscating an email address according to one embodiment. While FIG. 18is specific to obfuscating an email address, it should be understoodthat similar operations apply to obfuscate other elements that may beincluded in the requested resource (e.g., phone numbers, emailaddresses, instant messenger IDs, street addresses, links to otherwebsites, birthdates, social security numbers, IP addresses, credit cardnumbers, account usernames, etc.).

At block 1810, the proxy server 120 determines whether the email addressis within an area of the HTML page that is safe to modify. Generally, anarea that is safe to modify is an area of the HTML page in which theobfuscation script can execute. Examples of areas in which theobfuscation script cannot execute (and are thus not safe areas tomodify) include the header of the page, a comment, or another script. Inone embodiment, the resource modification module 275 tracks the currentstate of the HTML page to determine whether it is safe to modify theHTML page. HTML is made up of states that create text and markup. Markupis the content that either is not displayed to the user directly, orprovides the formatting for the content that is displayed to the user.For example, in an HTML page the contents of the <HEAD> section ismarkup. Any content in that section is not rendered on the page to theuser. Similarly, any content within the less-than and greater-thancharacters forms a tag. These tags then form markup which formats thepage to the user. Finally, certain other tags generate markup which isnot displayed. For example, the <script><pre><code> and <!--(comment)-->tags all create areas between their beginning and ending (e.g.,</script></pre></code> and <//--!>) that are not directly displayed.Email addresses in areas of the page that are directly displayed may bereplaced with the obfuscation script that will, programmatically,generate the replaced email address. If the email address is in markupor some other area of the page that prevents the rendering of script,then the email address will not be replaced with the obfuscation script.Thus, if the email address is within the header of a page, a comment,another script, or somewhere else where a script cannot execute, thenflow moves to block 1815 where the email address is not scrambled. Ifthe email address is within an area of the HTML page that is safe tomodify, then flow moves to block 1820.

With reference to block 1815, in some embodiments, even if the emailaddress is not replaced with an obfuscation script, the email addresscan still be obfuscated in such a way that the email address cannoteasily be harvested. For example, the resource modification module 275can replace the email address by spelling the [@] symbol and the [.]symbol (e.g., modifying <user@example.com> with <user [at]example[dot]com>). Other ways of obfuscating the email address may alsobe used (e.g., replacing the ASCII characters with their digitalequivalent, creating an image of the email address and replacing theemail address with that image, etc.).

At block 1820, the proxy server 120 determines whether the email addressis part of the text of the HTML page that is displayed to the visitor.If the email address is not part of the text, then flow moves to block1830. If email address is part of the text, then flow moves to block1825 and the proxy server 120 replaces the email address with anobfuscation script which, when executed by the client device, generatesthe email address to be displayed to the visitor. Thus, when viewing thesource of the HTML page, which is typically what automated email addressharvesting programs use to harvest email addresses, the email address isreplaced with the script, which cannot be typically interpreted by anautomated email address harvesting program. Thus, the email address willbe protected from being harvested thus reducing the amount ofunsolicited email to that email address. Flow moves from block 1825 toblock 18 30.

At block 1830, the proxy server 120 determines whether the email addressis part of an anchor, link, or mailto element. If it is not, then flowmoves to block 1845 and the email address is not modified. However, ifthe email address is part of an anchor, link, or mailto element, thenflow moves to block 1835 and the proxy server 120 replaces the hypertextreference (href) with a link and a unique token (e.g., a unique stringof characters). Next, flow moves to block 1840 where the proxy server120 appends a script that, when executed by the client networkapplication, scans for the unique token and rewrites the token with thelink (e.g., mailto attribute) such that when clicked by the visitor,their email program will launch (if it is not already launched) and anew email message window will be created that is addressed to that emailaddress. Thus, when viewing the source of the HTML page, which istypically what automated email address harvesting programs use whenharvesting email addresses, the email address that is part of theanchor, link, or mailto element will be replaced with the script, whichcannot be interpreted by most automated email address harvestingprograms. Thus, the email address will be protected from being harvestedthereby reducing the amount of unsolicited email to that email address.

Referring back to FIG. 17, if at block 1745 it is determined that thetoken is not a scramble type, then flow moves to block 1755. At block1755, the proxy server 120 determines whether the token is a server sidedefined modification (SSDM) token, defined by a web administrator of therequested domain. If it is, then flow moves to block 1760 where theproxy server 120 reads the modification rules associated with the tokenand acts accordingly. Flow moves from block 1760 back to block 1730.

While FIG. 17 was described with reference to replacing structured data(an email address) with an obfuscation script, some client networkapplications are incapable of executing the script or have disabledscripts from being executed. In one embodiment, if the obfuscationscript cannot execute, the email address simply will not be displayed onthe rendered page. There are multiple ways of determining whether aclient network application is incapable of executing scripts or hasdisabled scripts from being executed (e.g., if a script was known to beadded to a page and was not called, if content wrapped in a <noscript>tag was called, a user-agent is of a type that does not have scriptingenabled, etc.). By way of example, in some embodiments, the resourcemodification module 275 modifies the page to include a <noscript> tag(if one does not exist in the original page) that is used to provide analternative mechanism for handling structured data for those visitorsthat have disabled scripts from executing in their client networkapplication or are using a client network application that does notsupport scripts. The content included in the <noscript> tag for handlingstructured data may be different in different embodiments. For example,the content within the <noscript> tag can include an obfuscationmechanism (e.g., spelling the symbols of the email address, replacingthe ASCII characters with their digital equivalent, replacing the emailof the email address with an image of the email address, etc.). Asanother example, the content within the <noscript> tag can includeinformation that the email address has been removed, a link to anotherpage may (selecting the link may redirect the visitor to pass anadditional test such as a CAPTCHA before displaying the email address tothe visitor), or a contact form whose contents are relayed to the emailaddress which may also require a successful CAPTCHA before relaying thedata input into the contact form.

FIG. 19 is a flow diagram illustrating exemplary operations forprocessing server side defined modification tokens according to oneembodiment. In some embodiments, each server side defined modificationtoken is defined with a set of one or more default modification rulesset by the customers. For example, each customer may set defaultmodification rules through the system server 125 that specify theconditions on which a visitor will be subject to the rule and how tomodify the content represented by the token if the rule is triggered.These default rules may be different for different domains and/or filesof the domain. These default rules may also be overridden by includingrules within a SSDM token. Examples of modification rules include thefollowing: modify the content for certain visitor IP addresses (e.g., IPaddresses of search engines or other crawlers, modify the content for IPaddresses of certain location(s) (e.g., countries)), modify the contentfor visits at a particular time of day; modify the content for IPaddresses of a particular range; modify the content when a threat scoreof the visitor is above a defined threshold, and modify the content foridentified operating systems of visitors. Customers may definemodification rules that override the default rules, and may include themwithin the SSDM token. Thus, at block 1910, the proxy server 120determines whether the SSDM token itself includes one or moremodification rules. If the SSDM token includes a modification rule, thenflow moves to block 1915; otherwise flow moves to block 1920. At 1915,the rules are read from within the token and override the default rules.At 1920, the default modification rules are determined for theparticular SSDM token. Flow moves from both 1915 and 1920 to 1925.

At block 1925, the proxy server 120 reads the fingerprint of thevisitor's client network application. Flow then moves to block 1930where the proxy server 120 determines whether, based on the visitorcharacteristics, whether the visitor triggers application of one or moreof the modification rules for the SSDM token. If the visitor does nottrigger application of a rule, then flow moves to block 1935 where thetoken is ignored and processing continues (e.g., flow moves back toblock 1730 of FIG. 17). If the visitor does trigger application of oneor more rules, then flow moves to block 1940.

According to one embodiment, each SSDM token indicates an action totake, where the action may be specified in the correspondingmodification rule. An exclude action will exclude the contentrepresented by the token from the response to the visitor. An obfuscateaction will replace the content represented by the token with a script,which when executed, generates the replaced content to prevent automatedbots from easily being able to read that content. At block 1940, theproxy server 120 determines whether the token indicates an excludeaction. If it is, then flow moves to block 1945; otherwise flow moves toblock 1950.

At block 1945, the proxy server 120 removes the content represented bythe SSDM token. For example, the HTML between the opening and closingtag of the token is removed. Flow then moves to block 1955 whereprocessing continues (e.g., flow moves back to block 1730 of FIG. 17).

At block 1950, the proxy server 120 determines whether the SSDM tokenindicates an obfuscate action. If it does not, then flow moves to block1960 where alternative action is taken (e.g., no action is performed onthe content represented by the token and processing continues, or avisitor defined action is performed on the content). If the tokenindicates an obfuscate action, then flow moves to block 1965 where theproxy server 120 replaces the content represented by the token with ascript which, when executed, generates the content that was replaced.The script operates such that human users will be able to read thecontent when the script is executed, yet it is difficult for automatedbots to read that content. Thus, defining portions of the content aswrapped in a SSDM token with an obfuscate action effectively hides thatcontent for those visitors that trigger the modification rule. Flow thenmoves to block 1955 where processing continues.

Referring back to FIG. 17, if the token is not a SSDM token, then flowmoves to block 1758 where the proxy server 120 determines whether thetoken is an advertisement token. In one embodiment, to determine whetherthe HTML includes advertisement tokens, the proxy server 120 examinesthe HTML using a regular expression or other standard search techniquein order to find keywords that indicate the presence of advertising. Anadvertising keyword is a string of characters for an advertisingnetwork, which may be specific to an advertising network, that indicatesthe existence of an advertisement. In some embodiments, advertisingkeywords for one or more advertising networks are stored at the proxyserver 120 (e.g., in an advertisement data structure) or available tothe proxy server 120 from a remote database or other device. Sometimeafter an advertisement keyword is found, the proxy server 120 correlatesthe keyword with the advertising network in order to determine the sizeand shape of the advertisement. For example, the advertisement mayinclude location keywords such as “height” and “width” followed by theirvalues. When possible, the proxy server 120 may also determine otherattributes of the existing advertising (e.g., color, border, type,etc.).

If a particular advertising network does not provide information aboutthe advertisement within the HTML, the proxy server 120 uses analternative technique in order to determine the size and shape of theadvertisement. For example, in one embodiment, the proxy server 120loads the URL of the advertisement in a client network application ofthe proxy server, which can be done either dynamically as the page isreturned to a visitor or sometime after the page is returned to avisitor, with the result stored and associated with the particular pageand location of the advertisement. If the loaded element is an image,flash object, video object, or other type of displayed object, theclient network application of the proxy server 120 calculates theobject's height and width based on the metadata within the loadedobject. The height and width of the object would then be stored in adata structure (e.g., an advertisement database, which may be specificto the proxy server 120 or common to a group of proxy servers of theservice) and associated with the page where the advertisement appears.If the page is loaded again in the future, the proxy server 120 canaccess the advertisement data structure to determine the height andwidth of the advertisement displayed on the page. The proxy server 120may periodically resample the underlying advertising image in order toensure that the size has remained the same.

In other embodiments, instead of searching for advertising keywordsdefined by advertising networks, the proxy server 120 searches the HTMLpage for advertising keyword tags defined by the service. For example, aservice defined keyword tag instructs the proxy server 120 to include anadvertisement of a particular size and shape on the page at a particularposition of the page. Thus, rather than replacing an existingadvertisement, the proxy server would simply insert an advertisement ata defined position in the page. For example, the tag<!--INSERT_AD_HERE:728×90//--> indicates to the proxy server 120 toinsert an advertisement at the location of the tag of the defined size.

If an advertisement has been detected, then flow moves to block 2125 ofFIG. 21, otherwise flow moves to block 1762 and the proxy server logsthe token type (which may be unknown) including any variables or rulesthat have been included in that token. Flow then moves back to block1730.

Referring back to block 1710, if the requested resource is not an HTMLpage, then flow moves to block 1765 where the proxy server determineswhether the requested resource is a type that can potentially be harmfulto the client device (e.g., capable of containing malicious code (e.g.,virus, worm, malware, etc.)). For example, executable files arepotentially harmful to the client device. If the requested resource isnot potentially harmful, then flow moves to block 1770 where therequested response is transmitted to the client device. However, if therequested resource is potentially harmful, then flow moves to block1775.

At block 1775, the proxy server 120 scans the requested resource forthreats (e.g., viruses, worms, malware, etc.) and flow moves to block1780. If a threat is not detected, then flow moves to block 1770 and therequested resource is transmitted to the client device. If a threat isdetected, however, then flow moves to block 1785 where alternativeaction is taken (e.g., the response is blocked and the visitor and/ordomain owner may be notified).

Adding Content to the Response

In some embodiments, the proxy server 120 adds content to the response162 before delivering it to a client device 110. By way of example, acustomer may configure the service to add content for only certaindemographics (e.g., operating system type, client network applicationtype, country of origin, time of day, number of times they havepreviously visited the site, etc.) and/or only for a certain percentageof visitors (the percentage being definable by the customer and/or theservice).

In some embodiments the proxy server 120 adds a trap email addressand/or a trap form to the response 162. A trap email address is an emailaddress that is not used for any real email and is unique to aparticular IP address and session (thus the email address will not beknown or valid to different sessions and/or visitors).

FIG. 20 is a flow diagram illustrating exemplary operations for addingtrap email address(es) and/or trap form(s) to the content of a responseaccording to one embodiment. At block 2010, the proxy server receives arequest for content. Flow then moves to block 2015, where the proxyserver 120 determines whether the requested resource is an HTML page. Ifit is not, then flow moves to block 2020 where content will not be addedand processing continues as normal (e.g., the requested resource will betransmitted to the client device). If the requested resource is an HTMLpage, then flow moves to block 2025 where the proxy server retrieves therequested HTML page. The retrieval of the HTML page may be from eitherthe cache 122 or from the appropriate origin server. Flow moves fromblock 2025 to block 2030.

At block 2030, the proxy server 120 retrieves one or more trap emailaddresses and/or one or more trap forms to add to the HTML page. In oneembodiment, the available trap email addresses and trap forms are storedin a trap database. Flow then moves from block 2030 to block 2035 wherethe HTML page is modified to include the trap email address(es) and/ortrap form(s) in such a way that they are hidden from human users whenviewing the rendered page but are capable of being read, captured,and/or used when scanning/viewing the source page.

The trap email address(es) and/or trap form(s) can be hidden from humanusers in a number of ways. For example, CSS can be used to mark aparticular portion of code as having display=none or display=hidden.Alternatively, CSS can be used to move the link off the page (<ahref=“http://www.example.com/” style=“position: absolute; left: −250px;top: −250px;”>Some Hidden Link</a>). A tag can be included with nocontents (e.g., <a href=“http://www.example.com”></a>). Content can beincluded in a comment (e.g., <!--<a href=“www.example.com”>Some HiddenLink</a><//--!>). A script can be used to hide a link after it has beenrendered. A link can be wrapped around a single-pixel, or extremelysmall, image. It should be understood that these are exemplarytechniques and other techniques may be used to include links in a pagethat is visible to bots but hidden from humans.

Thus, unless a human user views the source of the HTML page, the userwill not notice or know that the trap email address(es) and/or trapform(s) have been added to the content. However, bots, which scan thesource of the HTML page when operating, will be able to harvest theemail address(es) in the source and attempt to POST data through thetrap form(s) in the source.

Flow moves from block 2035 to block 2040 where the proxy server 120associates the added trap email address(es) and/or trap form(s) with thevisitor and the visit. For example, the proxy server 120 recordscharacteristics of the visitor (e.g., IP address, session information,etc.) and associates them with the trap email address(es) and/or trapform(s) that were added to the HTML page. Thus, the added trap emailaddress(es) and/or trap form(s) are unique to the visitor. Accordingly,if an email is received at an account corresponding to a trap emailaddress added to the content, there is a strong likelihood that thevisitor was responsible for sending that email. Similarly, if data hasbeen input using a trap form that was added to the content, there is astrong likelihood that the visitor was responsible for that input. Flowthen moves to block 2045 where the modified HTML page is transmitted tothe visitor.

In some embodiments, the proxy server 120 adds and/or changesadvertisements to the response 162. FIG. 21 is a flow diagramillustrating exemplary operations for adding or changing advertisementsto requested resources according to one embodiment. In some embodiments,the customers choose whether pages from their domain are capable havingtheir advertisements changed and/or changing other content on the pagesuch as adding links around keywords (e.g., search keywords of highvalue) and/or replacing token links to add affiliate marketing programcodes. In one embodiment, the operations described in FIG. 21 start fromblock 1758 of FIG. 17 (e.g., an advertisement token has been detected inthe HTML content).

At block 2125 (an advertisement has been detected), the proxy server 120determines whether it is appropriate to replace the advertisement. If itis appropriate, then flow moves to block 2130, otherwise flow moves toblock 2135. In one embodiment, the proxy server 120 determines thisbased on whether it has access to an alternative advertisement of thesame size and whether the replacement advertisement would generate morerevenue (e.g., to the service and/or the customer) than the originaladvertisement. For example, the proxy server 120 accesses anadvertisement data structure to determine whether there are any otheradvertisements available (e.g., as stored in the advertisement datastructure) that match the height and width of the existingadvertisement. If there are existing advertisements available thatmatch, the system then checks the price that the advertisement wouldgenerate. This price may be determined based on a number of factorsincluding the characteristics of the particular visitor to the page(e.g., the geographic location, any information about the demographicprofile of the visitor (e.g., operating systems that cost more thanothers may indicate that the visitor has a relatively high income),etc.), the date and time of the visit, as well as the particular websiteor webpage being visited.

In one embodiment, the proxy server 120 replaces an existingadvertisement only if the revenue that would generated for displayingthe replacement advertisement is more than the revenue that would begenerated for displaying the existing advertisement. In anotherembodiment, the proxy server 120 replaces an existing advertisement atanytime the revenue that would be generated by displaying thereplacement advertisement was more than a threshold amount, which may beset by the service or by the customer.

At block 2130 (it is appropriate to replace the advertisement), theproxy server 120 replaces the advertisement. In one embodiment, theproxy server 120 modifies the HTML by deleting the reference to theoriginal advertisement including any links to the originaladvertisement. In another embodiment, the proxy server 120 does notremove the HTML but instead comments it out by adding an opening HTMLcomment tag before the advertisement object reference and link and afterthe object reference and link. In another embodiment of the invention,the proxy server 120 rewrites the HTML to add style tags (e.g.,style=“display:none;”) to the HTML elements in order to hide them frombeing displayed.

After the existing advertising object and link have been removed orhidden, the proxy server 120 replaces the advertising object and link.For example, the proxy server may, in one embodiment, insert a referenceto an HTML script object. The HTML script object may refer to a resourcethat is stored separately from the web page. Once loaded, the referencedscript object would call a command to modify the HTML to draw anadvertising object and link. In an alternative embodiment, the proxyserver 120 does not use a script object tag but, instead, directlyreferences an image, flash object, video, or some other object stored ina location separate from the web page. The proxy server would includethis object reference in the HTML and wrap an anchor tag around it inorder to create a link to the advertisement. For example, the proxyserver may insert:

-   -   <a href=“http://www.example.com/1234567890”><img        src=“http://www.example.com/images/ad1.jpg” height=“90px”        width=“728px”></a>        In either of the embodiments, when the HTML of the web page is        loaded, the visitor's client network application automatically        loads the references to the object references and displays the        advertisement on the page.

In an alternative embodiment of the invention, the data to create theimage, flash object, video, or other object may be embedded on the pageitself. This may be done to prevent ad blocking software running on avisitor's client device from excluding external advertisement referencesfrom loading. In such embodiments, the data to make up the image isinserted directly into the HTML of the page, and the visitor's clientnetwork application loads the raw byte data of an image or other objectwithout querying an external site or other reference. The data wouldthen be converted into a base form that could be displayed through HTML(e.g., base-64, base-16, base-10, etc.). A tag would be generatedincluding both the encoded byte data and the base it was encoded in. Ananchor tag would then be wrapped around the HTML object in order tocreate a link. The following is an example:

-   -   <a href=“http://www.example.com/1234567890”><img        src=“data:image/gif;base64,R01GOD1hEAAOALMAAOazToeHh0tLS/7LZv/Ojvb2        9t/f3//Ub//ge8WSLf/rhf/3kdbW1mxsbP//mf///yH5BAAAAAAALAAAAAAQAA4AAARe8L1Ekyky67QZ1hLnjM5UUde0ECwLJoExKcppV0aCc        GCmTIHEIUEqjg        aORCMxIC6e0CcguWw6aFjsVMkkIr7g77ZKPJjPZqIyd7sJAgVGoEGv2xsBxqNg        YPj/gAwXEQA7”></a>

Using this method, the advertisement image, flash object, video, orother object would be displayed without referencing a remote system.This avoids the possibility of the remote reference being blocked by anadvertising blocker on the visitor's client network application andensures that the advertisement is displayed.

Assigning Threat Scores for Visitors

The threat database 124 contains information that indicates whether avisitor poses a threat. The information may come from a variety ofsources including from customers of the service (e.g., webadministrators of the origin servers 130A-L), third party sources, andfrom the use of the service itself (e.g., from the proxy server 120). Insome embodiments, third party information may be used in setting threatscores for visitors. In such embodiments, third party scores forparticular IP addresses may be added to customer based scores for thoseIP addresses to create an overall threat score.

In some embodiments, the service provides functionality for customers ofthe service to report suspicious activity. For example, the serviceserver 125 provides an interface for the customers of the service toview visitor statistics for their website and report suspiciousvisitors.

FIG. 23 is a block diagram illustrating an exemplary threat reportinginterface 2310 according to one embodiment of the invention. The threatreporting interface 2310 shows visitor characteristics 2315 for recentvisitors (e.g., IP address, User-Agent, country of the visitor, previousvisits, pages visited, information posted) as reported by the proxyserver(s). The threat reporting interface 2310 includes the threatbutton 2325, which when selected, reports the corresponding visitor as athreat. The threat reporting interface 2310 also includes the not threatbutton 2330, which when selected, reports the corresponding visitor hasnot a threat. In some embodiments, the type of threat may also beprovided. For example, FIG. 24 illustrates an exemplary threat type form2410 which allows the type of threat (e.g., attack POST, excessivebandwidth, vulnerability, or other threat) to be selected and submittedthrough the submit button 2415.

FIG. 22 is a flow diagram illustrating exemplary operations for acustomer of the service to input threat information about one or morevisitors according to one embodiment. At block 2210, a customer logsinto the service server 125 (e.g., by providing their username andpassword). Flow then moves to block 2215, where the service server 125queries for the visitor statistics for the network resources of thecustomer. For example, the service server 125 queries the event logdatabase 126 for the visitor statistics of the customer. Flow then movesto block 2220, where the visitor statistics are displayed to thecustomer (e.g., displayed in an interface similar to the exemplaryinterface of FIG. 23). The displayed statistics may be sorted bycustomer preferences or a default value (e.g., most recent visitors,biggest threat, etc.). Flow then moves to block 2225, where the serviceserver 125 receives a selection from the customer that a visitor is athreat or is not a threat. Flow then moves to block 2230 where theservice server 125 records the customer selection in the threat database124. For example, if the customer indicated that a visitor is a threat,the service server 125 may populate the IP address of the visitor to oneor more restricted lists (e.g., the global restricted IP address listand/or the local restricted IP address list).

In some embodiments, the visitors are assigned a threat score, which isused to determine whether a visitor should be included on a restrictedlist. The threat score may depend on ratings from the customers of theservice. FIG. 25 is a flow diagram illustrating exemplary operations forusing customer defined threat information to assign threat scores tovisitors. According to one embodiment, the service server 125periodically performs the operations illustrated in FIG. 25. While theoperations of FIG. 25 are described with reference to the service server125, in other embodiments of the invention the proxy servers of theservice can perform the operations.

At block 2510, the service server 125 reads a visitor record from theevent log database 126 for an IP address and/or the global cookie, whichmay include rating(s) from customer(s) (e.g., threat or not a threat).Flow moves to block 2515 where the service server 125 determines whetherthe visitor has been rated by any customers. If no, then flow moves backto block 2510 where another visitor record is read (if appropriate). Ifyes, then flow moves to block 2520 where the service server 125 readsthe customer reputation score for each customer who rated the visitor. Acustomer reputation score indicates the relative trustworthiness of acustomer that is submitting visitor ratings. A higher customerreputation score indicates more trustworthiness (and thus more weight tothe visitor rating) than a lower customer reputation score. An exemplaryway of calculating a customer reputation score is described in FIG. 26.Flow next moves to block 2525 where the service server 125 adds thecustomer reputations together. Flow moves from block 2525 to block 2530.

At block 2530, the service server 125 determines whether the sum of thecustomer reputations is enough to create a global rating for the visitor(e.g., whether it the sum is above a customer reputations threshold). Itshould be understood that a relatively small value of the customerreputations sum may not be sufficient to create a valuable global rating(i.e., there may not be sufficient data to make a global judgment on thevisitor). The specific value of the customer reputations threshold maybe based on empirical evidence. If the sum of the customer ratings isenough to create a global rating, then flow moves to block 2535,otherwise flow moves back to block 2510 where another visitor record isread.

At block 2535, the customer reputation scores are tallied for allpositive ratings of the visitor. It should be understood that it ispossible that the visitor does not have a positive rating and thus nocustomer reputation scores to tally in block 2535. Next, flow moves toblock 2540 where the customer reputation scores are tallied for allnegative ratings of the visitor. It should be understood that it ispossible for the visitor to not have a negative rating, and thus nocustomer reputation scores to tally in block 2540. Flow then moves toblock 2545 where the negative rating tally is subtracted from thepositive rating tally to produce a result. Flow then moves to block 2550where the result is divided by the number of customer ratings for thevisitor. Flow then moves to block 2555.

At block 2555, the service server 125 determines whether the result isnegative. If it is, then flow moves to block 2560 where the visitor isrecorded as a threat (e.g., the IP address is placed on the globalrestricted IP address list) and the result (which may be normalized) isthe threat score for the visitor. The threat score may also be modifiedbased on other factors. If the result is not negative, then flow movesto block 2565 where the visitor is recorded as not a threat (e.g., ifincluded, the IP address is removed from the global restricted IPaddress list). Flow moves from block 2560 and 2565 back to block 2510.

FIG. 26 is a flow diagram illustrating exemplary operations forcalculating a customer reputation score according to one embodiment.FIG. 26 will be described in reference to the service server 125;however in other embodiments the operations described in reference toFIG. 26 can be performed by the proxy servers of the service. At block2610, the service server 125 reads a customer record for all thevisitors that the customer has rated. Flow then moves to block 2615where the service server 125 determines whether there are visitors thathave been rated that have not yet been accounted for in the customersreputation. In one embodiment, a bit or flag is set for each visitorregarding whether it has been accounted for in the customers reputation,which may expire after a certain amount of time. If there are ratedvisitors that have not yet been accounted for, flow moves to block 2620,otherwise flow moves back to block 2610 and another customer record isread (if appropriate).

At block 2620, the service server 125 reads a visitor rating that is notaccounted for in the customers reputation score. Flow then moves toblock 2625 where the service server 125 determines whether the rating ofthe visitor (e.g., threat or no threat) aligns with the community rating(e.g., determined through the operations described in FIG. 25). If thecustomer rating does not align with the community rating, then flowmoves to block 2635 where the customer reputation score is reduced by anamount. Thus, the customer's reputation score will be reduced when ithas rated a visitor that does not align with the community rating. Ifthe customer rating aligns with the community rating, then flow moves toblock 2630 where the customer reputation score is increased by anamount. Thus, the customer's reputation score will be increased when ithas rated a visitor that aligns with the community rating. Flow movesfrom block 2630 and 2635 back to block 2615.

As illustrated in FIG. 27, the computer system 2700, which is a form ofa data processing system, includes the bus(es) 2750 which is coupledwith the processing system 2720, power supply 2725, memory 2730, and thenonvolatile memory 2740 (e.g., a hard drive, flash memory, Phase-ChangeMemory (PCM), etc.). The bus(es) 2750 may be connected to each otherthrough various bridges, controllers, and/or adapters as is well knownin the art. The processing system 2720 may retrieve instruction(s) fromthe memory 2730 and/or the nonvolatile memory 2740, and execute theinstructions to perform operations described herein. The bus 2750interconnects the above components together and also interconnects thosecomponents to the display controller & display device 2770, Input/Outputdevices 2780 (e.g., NIC (Network Interface Card), a cursor control(e.g., mouse, touchscreen, touchpad, etc.), a keyboard, etc.), and theoptional wireless transceiver(s) 2790 (e.g., Bluetooth, WiFi, Infrared,etc.). In one embodiment, the client devices 110A-I, the service server125, the proxy server 120, the validating domain server 180, and/or theorigin servers 130A-L can take the form of the computer system 2700.

The techniques shown in the figures can be implemented using code anddata stored and executed on one or more computing devices (e.g., clientdevices, servers, etc.). Such computing devices store and communicate(internally and/or with other computing devices over a network) code anddata using machine-readable media, such as machine-readable storagemedia (e.g., magnetic disks; optical disks; random access memory; readonly memory; flash memory devices; phase-change memory) andmachine-readable communication media (e.g., electrical, optical,acoustical or other form of propagated signals—such as carrier waves,infrared signals, digital signals, etc.). In addition, such computingdevices typically include a set of one or more processors coupled to oneor more other components, such as one or more storage devices, userinput/output devices (e.g., a keyboard, a touchscreen, and/or adisplay), and network connections. The coupling of the set of processorsand other components is typically through one or more busses and bridges(also termed as bus controllers). The storage device and signalscarrying the network traffic respectively represent one or moremachine-readable storage media and machine-readable communication media.Thus, the storage device of a given computing device typically storescode and/or data for execution on the set of one or more processors ofthat computing device. Of course, one or more parts of an embodiment ofthe invention may be implemented using different combinations ofsoftware, firmware, and/or hardware.

In some embodiments, different aspects of the service are disabledduring periods of heavy load on particular proxy servers and/or theservice as a whole. For example, to disable the features, the zone filerecords for the domains may be changed such that DNS resolution requestsfor the domains owned by the domain owners 135A-L, which correspond withthe origin servers 130A-L respectively, resolve back to the appropriateorigin servers instead of the proxy server 120. This may occur on aproxy server by proxy server basis, globally across all proxy servers inthe service, or any combination thereof. In one embodiment, the DNS zonefile records are changed automatically without input from the customerdomain owners (e.g., the service server 125 records the original zonefile information when the customers initially changed their DNS zonefile records). It should be understood that once the zone file recordsare changed, requests will be transmitted directly to the origin webservers and thus the caching service, the threat analysis/blockingservice, analytical service, etc., that is provided by the service willnot be available.

While the flow diagrams in the figures show a particular order ofoperations performed by certain embodiments of the invention, it shouldbe understood that such order is exemplary (e.g., alternativeembodiments may perform the operations in a different order, combinecertain operations, overlap certain operations, etc.).

While the invention has been described in terms of several embodiments,those skilled in the art will recognize that the invention is notlimited to the embodiments described, can be practiced with modificationand alteration within the spirit and scope of the appended claims. Thedescription is thus to be regarded as illustrative instead of limiting.

What is claimed is:
 1. A method in a proxy server for responding toresource unavailable errors, comprising: receiving, from a clientdevice, a request for a network resource hosted at an origin server fora domain, wherein the request is received at the proxy server as aresult of a DNS (Domain Name System) request for the domain resolving tothe proxy server, and wherein the origin server is one of a plurality oforigin servers that belong to different domains that resolve to theproxy server and are owned by different entities; requesting the networkresource from the origin server; receiving a response from the originserver that indicates that the network resource is unavailable; andtransmitting a custom error page to the client device that indicatesthat the requested resource is unavailable.
 2. The method of claim 1,further comprising: determining that the requested network resource hasmoved to a different location; and including in the custom error page alink to the different location for the requested network resource. 3.The method of claim 2, wherein determining that the requested networkresource has moved to a different location includes matching a hash ofthe requested network resource with one of a plurality of hashes ofother network resources hosted by the origin server.
 4. The method ofclaim 1, wherein the custom error page is customized based on a countryof a visitor belonging to the request or a language of the visitor. 5.The method of claim 1, wherein the request includes a set of one or moreterms, and wherein the custom error page is customized based on the setof terms.
 6. The method of claim 1, wherein the custom error pageincludes a set of one or more links to other network resources availableon the domain.
 7. The method of claim 1, wherein the custom error pageincludes a set of one or more links to other network resources availableon a set of one or more domains other than the domain that hosts therequested network resource.
 8. A proxy server for responding to resourceunavailable errors, the proxy server comprising: a request moduleconfigured to receive requests from client devices for network resourceshosted at a plurality of origin servers for a plurality of domains andfurther configured to transmit each of those requests to the appropriateorigin server, wherein DNS (Domain Name System) requests for each of theplurality of domains resolve to the proxy server; a response andresource analyzer module configured to receive responses from theplurality of origin servers in response to the requests and determinewhether the responses indicate a resource unavailable error; anunavailable resource module coupled with the response and resourceanalyzer module that is configured to create a customized error page foreach response that indicates a resource unavailable error; and aresponse forming and transmission module coupled with the unavailableresource module that is configured to form and transmit the customizederror pages to the
 9. The proxy server of claim 8, wherein theunavailable resource module is configured to include in the customizederror page a link to a current location of a requested network resource.10. The proxy server of claim 8, wherein the unavailable resource moduleis configured to create the customized error page for each response thatindicates a resource unavailable error based on a country of a visitorbelonging to the corresponding request or a language of the visitor. 11.The proxy server of claim 8, wherein at least some of the requestsinclude a set of one or more terms, and wherein the unavailable resourcemodule is configured to base the customized error page for each of thoseresponses whose corresponding request includes a set of one or moreterms on those terms.
 12. The proxy server of claim 8, wherein theunavailable resource module is configured to include in the customizederror page for each of those responses a set of one or more links toother network resources.
 13. The proxy server of claim 12, wherein theset of links to the other network resources are in a same one of theplurality of domains as the requested network resource.
 14. Anon-transitory machine-readable storage medium that providesinstructions that, when executed by a processor of a proxy server, causesaid processor to perform operations comprising: receiving, from aclient device, a request for a network resource hosted at an originserver for a domain, wherein the request is received at the proxy serveras a result of a DNS (Domain Name System) request for the domainresolving to the proxy server, and wherein the origin server is one of aplurality of origin servers that belong to different domains thatresolve to the proxy server and are owned by different entities;requesting the network resource from the origin server; receiving aresponse from the origin server that indicates that the network resourceis unavailable; and transmitting a custom error page to the clientdevice that indicates that the requested resource is unavailable. 15.The non-transitory machine-readable storage medium of claim 14, whereinthe operations further comprise: determining that the requested networkresource has moved to a different location; and including in the customerror page a link to the different location for the requested networkresource.
 16. The non-transitory machine-readable storage medium ofclaim 15, wherein determining that the requested network resource hasmoved to a different location includes matching a hash of the requestednetwork resource with one of a plurality of hashes of other networkresources hosted by the origin server.
 17. The non-transitorymachine-readable storage medium of claim 14, wherein the custom errorpage is customized based on a country of a visitor belonging to therequest or a language of the visitor.
 18. The non-transitorymachine-readable storage medium of claim 14, wherein the requestincludes a set of one or more terms, and wherein the custom error pageis customized based on the set of terms.
 19. The non-transitorymachine-readable storage medium of claim 14, wherein the custom errorpage includes a set of one or more links to other network resourcesavailable on the domain.
 20. The non-transitory machine-readable storagemedium of claim 14, wherein the custom error page includes a set of oneor more links to other network resources available on a set of one ormore domains other than the domain that hosts the requested networkresource